#32Build a workflow engine (Temporal / Airflow / Cadence style)
A function that survives crashes, restarts, and re-deploys — and still finishes. Build a durable execution engine where workflow code is replayed deterministically from an event history, activities retry with exponential backoff, sagas compensate on failure, and the same workflow definition runs identically a year later. Internalize why 'just retry the cron job' breaks at the second step.

You have written a function that does a few things in order — charge a card, reserve some inventory, ship a package, send an email. You have written a cron job that retries something that failed. That is your starting point. By the end of this curriculum you will have built — conceptually, layer by layer — a Temporal/Cadence-style durable execution engine, and you will be able to reason about a real workflow: where its state lives, what happens to it when the process crashes mid-run, when a retry is safe, how it waits 30 days without holding a thread, and how to change its code a year after it started — and choose, for a given workload, whether a durable engine, a DAG scheduler, or a state machine is the right tool.

We start from one uncomfortable, concrete fact. Scene 1 runs ORDER #1001: ChargeCard($42)ReserveInventory(1 Widget)ShipPackageSendConfirmationEmail. The process crashes right after the card is charged but before the inventory is reserved. A plain function kept its progress (charged = true) only in RAM, and the crash erased it — so a naive cron retry re-runs from the top and charges the card a second time: $84. That is what "just retry the cron job breaks at the second step" actually looks like. Every later mechanism in the stack exists to plug a worse-placed crash, and we re-ask the same question against each one: did we double-charge, and does ORDER #1001 still finish?

This is a curriculum about a strict density-of-vocabulary discipline: across 13 scenes, each scene introduces at most two new terms, and the diagram carries the load. One running example — ORDER #1001 — is threaded through every scene: you watch its progress get wiped from RAM, then written to a durable event history, then replayed to resume at step 2, then quarantined inside activities, then survive a redeploy, a retry storm, a crash inside the charge, a 30-day wait, a cancel signal, a mid-order failure, an endless billing loop, and finally a code change deployed while the order was still sleeping. The same blue-is-workflow / red-is-activity coloring and the same append-only history strip appear in every scene so the arc reads as one continuous story.

Resist the urge to "describe Temporal." Build each layer because the previous layer's crash forced it, feel the failure mode each one prevents, and let the design canvas at the end push back on whether durable execution was even the right tool.

Reading: Temporal — Durable Execution / Temporal Service concept page (the plain-English definition + the engine↔worker split): docs.temporal.io/temporal · Temporal — Workflow Definition, the determinism constraints (what you may NOT do in workflow code and why replay demands it): docs.temporal.io/workflow-definition · Temporal — Activities (the activity boundary, at-least-once semantics, why idempotency is the developer's job): docs.temporal.io/activities · Temporal — Retry Policies (the concrete backoff defaults: 1s, ×2, cap, jitter, non-retryable): docs.temporal.io/encyclopedia/retry-policies · Temporal — Versioning / patching (the longest-running-deploy problem, before you ever ship a v2): docs.temporal.io/develop/go/versioning · Garcia-Molina & Salem — 'Sagas', ACM SIGMOD 1987 (the origin of compensating transactions; compensation restores an acceptable approximation, not the exact prior state): dl.acm.org/doi/10.1145/38713.38742 · Greg Young / Martin Fowler — Event Sourcing (the pattern that underpins workflow history: the log is truth): martinfowler.com/eaaDev/EventSourcing.html · SE Radio 596 — Maxim Fateev on Durable Execution with Temporal (the creator, ex-AWS SWF / Uber Cadence, explaining the model conversationally): se-radio.net/2023/12/se-radio-596 · Uber Engineering — Cadence overview (Fateev & Abbas; the design rationale for durable, long-running business logic): uber.com/en/blog/open-source-orchestration-tool-cadence-overview · AWS Step Functions — developer guide (contrast the JSON state-machine / ASL model with code-as-workflow): docs.aws.amazon.com/step-functions/latest/dg/welcome.html · Apache Airflow — core concepts (DAG scheduling + task-level retry, and why it is NOT replay-deterministic): airflow.apache.org/docs/apache-airflow/stable/core-concepts · Azure Durable Functions — concepts (the replay model expressed via generators/yield; a useful third data point): learn.microsoft.com/azure/azure-functions/durable/durable-functions-overview · Stripe — Idempotency keys (the canonical real-world idempotency-key pattern; pair it with the double-charge scene): docs.stripe.com/api/idempotent_requests
durable execution: a function whose progress survives crashes, reboots, and redeploys — not just an in-memory call
the event history is the source of truth: append every step's result to a durable log, never trust RAM
event sourcing: rebuild current state by folding the log of events, instead of saving variables directly
deterministic replay: re-run the workflow code against the recorded history to resume — so the code must be pure
activities are the boundary to non-determinism: side effects (RPC, DB, time, randomness) run once and the RESULT is recorded
task queues + workers that PULL: the engine is decoupled from the worker, so a redeploy is just 'no worker for a moment'
retries with exponential backoff + per-activity policies (max attempts, backoff coefficient, jitter, non-retryable errors)
idempotency keys: the real defense against the double-charge under at-least-once delivery
durable timers as first-class events: 'sleep 30 days' survives ten restarts because the timer lives in history, not a thread
signals & queries: a running workflow is an addressable actor you can message and read
the saga pattern: compensating actions undo completed steps in reverse on partial failure (refund, not rollback)
child workflows + ContinueAsNew: compose sub-units and keep an endless workflow's history bounded
versioning workflows safely: a year-old in-flight execution must still replay identically against today's code