Arqly
Sign in
Build a workflow engine (Temporal / Airflow / Cadence style) (13 scenes)
Scene 01 · The crash that charges you twice
A plain function keeps its progress in RAM, so a crash after step one erases it — and a naive cron retry re-runs from the top and charges the card a second time.
Scene 01
The crash that charges you twice
Diagram
The center is ORDER #1001 as four steps inside one process box. The RAM panel shows the live progress variables — and that panel is wiped to blank the instant the skull hits, because a plain function's state only ever lived in memory. The $ meter is the customer's statement. 'Durable execution' is the property we WANT: progress that survives the skull. 'At-least-once delivery' is the only promise the network makes: a call may run 0, 1, or many times, and a timeout never tells you which.
Sources
ORDER #1001 — one plain function, four stepsred = activity (runs once, for real) · state lives in RAM → gone on crashPROCESSruns the four-step function top-to-bottomChargeCard$42✓ ran onceReserveInven…✓ ran onceShipPackage✓ ran onceSendConfirma…✓ ran onceRAMstate lives herecharged = falsestep = 0the only record of progressSTATEMENT · ORDER #1001$42charged once · cleanno retrythe run finished — nothing to retryat-least-once delivery — a call may run 0, 1, or many timesClean run: four steps, $42 once. But the only record of progress lives in RAM.
RAM: the live progress (charged=true, step=2). This is the ONLY record that step 1 ran.
Start with something that looks completely ordinary. ORDER #1001 is one plain function with four steps in order: ChargeCard($42), ReserveInventory, ShipPackage, SendConfirmationEmail. Run it once and it finishes clean — the statement reads $42, and the RAM panel ticks its progress as it goes: charged=true, then step=2, 3, 4. Now watch the same function the way real systems actually behave: the process can die at any moment. We'll drop a crash right after ChargeCard succeeds. The instant the skull hits, look at the RAM panel — it wipes blank. The only record that step 1 ever ran lived in that memory, and the crash erased it. The property this function is MISSING — progress that survives a crash, a reboot, a redeploy — is called **durable execution**. A plain in-memory call does not have it. Watch the clean run first, then the crash that wipes the proof.
Implementation
fulfillOrder
a plain function — progress lives only in local variables
1def fulfillOrder(order):
2    charged = False        # in RAM, dies with the process
3    chargeCard(order, $42)  # moves real money at the card network
4    charged = True
5    reserveInventory(order)
6    shipPackage(order)
7    sendConfirmationEmail(order)
cron.retry
the unfinished order is re-run from line one
1# every minute: find orders that never finished
2for order in db.ordersNotMarkedDone():
3    # no memory of which steps already ran —
4    # the only such record lived in RAM, now gone
5    fulfillOrder(order)   # starts again at chargeCard
Not sure what to ask? Tap a question — the staff engineer answers in the chat panel.
NextThe event history is the source of truth