The saga: compensate partial failure in reverse

Because you can't wrap steps across separate services in one transaction, a saga gives each step a compensating action that semantically undoes it, and on a mid-order failure it runs the compensations for completed steps in reverse order — a refund, not a rollback to a prior byte-state.

Previously

The saga safely backs ORDER #1001 out of partial work, refunding and releasing in reverse. But we've kept ORDER #1001 as ONE giant workflow with one growing history — and real orders split across warehouses, run for months, and bill forever. A single history can't grow without bound, and one monolithic workflow is hard to reason about. How do we compose and how do we run forever?

Scene 09

The saga: compensate partial failure in reverse

Watch
Try it
Predict
Capture

What to watch for

ORDER #1001 has four steps that live on four different services: ChargeCard ($42), ReserveInventory (one Widget), ShipPackage, then SendConfirmationEmail. Watch them stack into a tower as each one completes — first the charge clears, then the Widget is reserved. Then ShipPackage can't complete: the warehouse is out of stock, permanently. Now you're stuck in the worst place: you've already taken $42 and reserved a Widget, but you can't ship — and there is no single database transaction wrapping these four separate services that you could simply roll back. Notice the ghost block sitting next to each completed step — RefundCard beside ChargeCard, ReleaseInventory beside ReserveInventory. That's a sequence of steps where every step carries its own undo. The systems-design name for modeling a long operation this way — local steps, each with a compensating action — is a saga (Garcia-Molina & Salem, 1987): the practical substitute for a distributed transaction when you can't get one.

ORDER #1001's steps stack as a tower.
A 'saga' is a sequence of steps, each paired with a compensating action that semantically undoes it (RefundCard undoes ChargeCard; ReleaseInventory undoes ReserveInventory).
When a step fails partway, the tower un-builds in REVERSE order, running the compensators for the steps that already completed.
'Compensation (semantic undo)' is the key point: a refund is not a rollback — the history shows both the charge AND the refund; it restores an acceptable approximation, never the exact prior byte-state.
Compensators can fail too, so they must be retryable.

Implementation

Highlighted lines are the ones running in the diagram right now.

OrderWorkflow.run

run the steps forward, compensate on partial failure

1def run(order):
2    done = []                      # completed steps, in order
3    for step in [Charge, Reserve, Ship, Email]:
4        try:
5            step.do(order)         # an activity on its service
6            done.append(step)
7        except StepFailed:
8            compensate(done)       # reverse-undo what completed
9            raise

Saga.compensate

undo the completed steps, newest first

1def compensate(done):
2    for step in reversed(done):    # backward recovery
3        if step.compensator is None:
4            continue               # no clean undo (Ship/Email)
5        compensateStep(step.compensator)

Saga.compensateStep

a compensator is just another retryable activity

1def compensateStep(comp):
2    delay = 1.0                    # initialInterval
3    while True:
4        try:
5            comp.do(idempotency_key)   # semantic undo, dedup-safe
6            return                     # append Compensated(comp)
7        except ActivityFailed:
8            sleep(delay)
9            delay *= 2.0               # backoffCoefficient

Sources

paperSagas

Where this sits in Build a workflow engine (Temporal / Airflow / Cadence style)

Scene 09 of 13, in the Scale & undo act — Sagas compensate; children + ContinueAsNew.. You can't wrap steps across services in one transaction; a saga gives each step an undo and runs the compensators for completed steps in reverse — a refund, not a rollback.

Up next. Sagas handle one order failing, but a real order can split across warehouses and a subscription can bill forever. The answers are composition — a parent workflow starting child workflows it awaits — and a way to keep an endless workflow's history from growing without bound by restarting it fresh while keeping its ID. That restart is ContinueAsNew.

All 13 scenes in Build a workflow engine (Temporal / Airflow / Cadence style) · Every curriculum

Built with Arqly

Every scene in Build a workflow engine (Temporal / Airflow / Cadence style) builds on the one before it.All 13 Build a workflow engine (Temporal / Airflow / Cadence style) scenes