Idempotency keys: the last hole in the double-charge

Build a workflow engine (Temporal / Airflow / Cadence style) (13 scenes)

Scene 6.5 · Idempotency keys: the last hole in the double-charge

An activity can run twice if it succeeds then crashes before recording — a stable idempotency key lets the downstream recognize the repeat and refuse the second charge.

Previously

Retries let a transient 503 self-heal — but a retry also means an activity can RUN more than once. Picture the worst crash yet: the worker charges the card, then dies before recording the result. The engine, seeing no result, retries — and charges again. History and replay can't help, because the effect already happened outside the recorded boundary. What stops THAT double-charge?

Scene 6.5

Idempotency keys: the last hole in the double-charge

Watch

Diagram

The strip is ChargeCard's own activity timeline: request sent → money moved → result recorded. Between the last two sits a red CRASH WINDOW. A crash there is the worst case — the card was already charged, but the engine never saw the result, so it retries and charges again. The 'idempotency-key' chip on the request is a stable label (order-1001-charge) the CALLER attaches; the payment API's DEDUP STORE recognizes a repeat carrying that key and returns the original result instead of charging twice — the caller makes it, the server enforces it. The banner names the deal: the engine gives you effectively-once workflow LOGIC, but activity EFFECTS are at-least-once, so closing this hole for money-moving steps is your job. The customer's statement reads $42 charged once, or $84 double-charged.

Sources

↑ the CRASH WINDOW: money moved, but no result recorded yet

↑ effectively-once workflow LOGIC, but at-least-once activity EFFECTS

← the idempotency-key chip: a stable label the caller attaches so the receiver can spot a repeat

We already moved every side effect into an activity, so replay never re-charges the card. But zoom all the way into the ChargeCard activity itself. It has three moments: the request is sent, the money actually moves at the card network, and finally the result is recorded back in history. The danger lives in the gap between the last two. If the worker crashes AFTER the money moved but BEFORE the result was recorded — the **CRASH WINDOW**, shaded red — the engine sees no recorded result and does exactly what we taught it to: it retries the activity. The money already moved once; the retry moves it again. This is the one hole history and replay cannot reach, because the effect happened outside the recorded boundary. The banner above the strip names the deal you actually get: the engine gives you **effectively-once** workflow LOGIC — your code's command is never re-issued — but activity EFFECTS are **at-least-once**, meaning the real-world charge can apply more than once. The only lever that closes this is the chip on the request: an **idempotency key** — a stable label the caller attaches (order-1001-charge) so the receiver can recognize a repeat and refuse to charge twice. Drop the skull into the red window and watch the statement.

Implementation

Engine.runActivity

delivery is at-least-once: no recorded result → run again

1def runActivity(activity, input):
2    while True:
3        try:
4            result = activity(input)  # money moves here
5            history.record(result)    # result recorded here
6            return result
7        except NoResultReported:
8            # crash landed after the call, before record
9            sleep(backoff())          # then retry the call

Activity.ChargeCard

the caller stamps a stable label on the request

1def charge_card(order):
2    # key is derived from data that survives a crash,
3    # so the retry carries the SAME label as the first try
4    key = idempotency_key(order)   # 'order-1001-charge'
5    return payment_api.charge(
6        amount = order.total,
7        idempotency_key = key,      # null when off
8    )

PaymentAPI.charge

the downstream dedup store enforces single-application

1def charge(amount, idempotency_key):
2    if idempotency_key in dedup_store:
3        # repeat recognized: return the first outcome
4        return dedup_store[idempotency_key]
5    result = card_network.move(amount)  # applies effect
6    if idempotency_key is not None:
7        dedup_store[idempotency_key] = result
8    return result

Not sure what to ask? Tap a question — the staff engineer answers in the chat panel.

PreviousRetries and exponential backoff NextDurable timers: sleep 30 days on zero compute