Build a workflow engine (Temporal / Airflow / Cadence style)

A function that survives crashes, restarts, and re-deploys — and still finishes. Build a durable execution engine where workflow code is replayed deterministically from an event history, activities retry with exponential backoff, sagas compensate on failure, and the same workflow definition runs identically a year later. Internalize why 'just retry the cron job' breaks at the second step.

SavedSaved on this device

The problem

A function that survives crashes, restarts, and re-deploys — and still finishes. Build a durable execution engine where workflow code is replayed deterministically from an event history, activities retry with exponential backoff, sagas compensate on failure, and the same workflow definition runs identically a year later. Internalize why 'just retry the cron job' breaks at the second step.

You have written a function that does a few things in order — charge a card, reserve some inventory, ship a package, send an email. You have written a cron job that retries something that failed. That is your starting point. By the end of this curriculum you will have built — conceptually, layer by layer — a Temporal/Cadence-style durable execution engine, and you will be able to reason about a real workflow: where its state lives, what happens to it when the process crashes mid-run, when a retry is safe, how it waits 30 days without holding a thread, and how to change its code a year after it started — and choose, for a given workload, whether a durable engine, a DAG scheduler, or a state machine is the right tool.

We start from one uncomfortable, concrete fact. Scene 1 runs ORDER #1001: ChargeCard($42) → ReserveInventory(1 Widget) → ShipPackage → SendConfirmationEmail. The process crashes right after the card is charged but before the inventory is reserved. A plain function kept its progress (charged = true) only in RAM, and the crash erased it — so a naive cron retry re-runs from the top and charges the card a second time: $84. That is what "just retry the cron job breaks at the second step" actually looks like. Every later mechanism in the stack exists to plug a worse-placed crash, and we re-ask the same question against each one: did we double-charge, and does ORDER #1001 still finish?

This is a curriculum about a strict density-of-vocabulary discipline: across 13 scenes, each scene introduces at most two new terms, and the diagram carries the load. One running example — ORDER #1001 — is threaded through every scene: you watch its progress get wiped from RAM, then written to a durable event history, then replayed to resume at step 2, then quarantined inside activities, then survive a redeploy, a retry storm, a crash inside the charge, a 30-day wait, a cancel signal, a mid-order failure, an endless billing loop, and finally a code change deployed while the order was still sleeping. The same blue-is-workflow / red-is-activity coloring and the same append-only history strip appear in every scene so the arc reads as one continuous story.

Resist the urge to "describe Temporal." Build each layer because the previous layer's crash forced it, feel the failure mode each one prevents, and let the design canvas at the end push back on whether durable execution was even the right tool.

What you will cover

durable execution: a function whose progress survives crashes, reboots, and redeploys — not just an in-memory call
the event history is the source of truth: append every step's result to a durable log, never trust RAM
event sourcing: rebuild current state by folding the log of events, instead of saving variables directly
deterministic replay: re-run the workflow code against the recorded history to resume — so the code must be pure
activities are the boundary to non-determinism: side effects (RPC, DB, time, randomness) run once and the RESULT is recorded
task queues + workers that PULL: the engine is decoupled from the worker, so a redeploy is just 'no worker for a moment'
retries with exponential backoff + per-activity policies (max attempts, backoff coefficient, jitter, non-retryable errors)
idempotency keys: the real defense against the double-charge under at-least-once delivery
durable timers as first-class events: 'sleep 30 days' survives ten restarts because the timer lives in history, not a thread
signals & queries: a running workflow is an addressable actor you can message and read
the saga pattern: compensating actions undo completed steps in reverse on partial failure (refund, not rollback)
child workflows + ContinueAsNew: compose sub-units and keep an endless workflow's history bounded
versioning workflows safely: a year-old in-flight execution must still replay identically against today's code

Reading

Temporal — Durable Execution / Temporal Service concept page (the plain-English definition + the engine↔worker split): docs.temporal.io/temporal
Temporal — Workflow Definition, the determinism constraints (what you may NOT do in workflow code and why replay demands it): docs.temporal.io/workflow-definition
Temporal — Activities (the activity boundary, at-least-once semantics, why idempotency is the developer's job): docs.temporal.io/activities
Temporal — Retry Policies (the concrete backoff defaults: 1s, ×2, cap, jitter, non-retryable): docs.temporal.io/encyclopedia/retry-policies
Temporal — Versioning / patching (the longest-running-deploy problem, before you ever ship a v2): docs.temporal.io/develop/go/versioning
Garcia-Molina & Salem — 'Sagas', ACM SIGMOD 1987 (the origin of compensating transactions; compensation restores an acceptable approximation, not the exact prior state): dl.acm.org/doi/10.1145/38713.38742
Greg Young / Martin Fowler — Event Sourcing (the pattern that underpins workflow history: the log is truth): martinfowler.com/eaaDev/EventSourcing.html
SE Radio 596 — Maxim Fateev on Durable Execution with Temporal (the creator, ex-AWS SWF / Uber Cadence, explaining the model conversationally): se-radio.net/2023/12/se-radio-596
Uber Engineering — Cadence overview (Fateev & Abbas; the design rationale for durable, long-running business logic): uber.com/en/blog/open-source-orchestration-tool-cadence-overview
AWS Step Functions — developer guide (contrast the JSON state-machine / ASL model with code-as-workflow): docs.aws.amazon.com/step-functions/latest/dg/welcome.html
Apache Airflow — core concepts (DAG scheduling + task-level retry, and why it is NOT replay-deterministic): airflow.apache.org/docs/apache-airflow/stable/core-concepts
Azure Durable Functions — concepts (the replay model expressed via generators/yield; a useful third data point): learn.microsoft.com/azure/azure-functions/durable/durable-functions-overview
Stripe — Idempotency keys (the canonical real-world idempotency-key pattern; pair it with the double-charge scene): docs.stripe.com/api/idempotent_requests

About Build a workflow engine (Temporal / Airflow / Cadence style)

Difficulty: advanced
Time: about 90 minutes
Stages: 9
Topic: Consensus, Coordination & Durable Execution

How this problem is worked

Nine stages, from what the thing is for to how it compares with the real implementations. Each asks one question, and the simulator runs the architecture you draw against the requirements you wrote.

01Purpose & invariantsWhat is this for, and what must always be true of it?
02Workload characterizationWho writes, who reads, and in what shapes?
03Data model & on-disk formatWhat does the data look like at rest?
04Core algorithmsHow do the write path and the read path actually work?
05Distribution & replicationHow does this scale out and survive losing a machine?
06Consistency & correctnessUnder concurrency and failure, what is guaranteed?
07Failure modes & recoveryWhat actually happens when each part fails?
08Operational characteristicsCan a human run this at three in the morning?
09Trade-offs & comparisonWhere does this sit against the alternatives?

Primary sources for this problem

Temporal — Durable Execution / Temporal Service concept page (the plain-English definition + the engine↔worker split): docs.temporal.io/temporal
Temporal — Workflow Definition, the determinism constraints (what you may NOT do in workflow code and why replay demands it): docs.temporal.io/workflow-definition
Temporal — Activities (the activity boundary, at-least-once semantics, why idempotency is the developer's job): docs.temporal.io/activities
Temporal — Retry Policies (the concrete backoff defaults: 1s, ×2, cap, jitter, non-retryable): docs.temporal.io/encyclopedia/retry-policies
Temporal — Versioning / patching (the longest-running-deploy problem, before you ever ship a v2): docs.temporal.io/develop/go/versioning
Garcia-Molina & Salem — 'Sagas', ACM SIGMOD 1987 (the origin of compensating transactions; compensation restores an acceptable approximation, not the exact prior state): dl.acm.org/doi/10.1145/38713.38742
Greg Young / Martin Fowler — Event Sourcing (the pattern that underpins workflow history: the log is truth): martinfowler.com/eaaDev/EventSourcing.html
SE Radio 596 — Maxim Fateev on Durable Execution with Temporal (the creator, ex-AWS SWF / Uber Cadence, explaining the model conversationally): se-radio.net/2023/12/se-radio-596
Uber Engineering — Cadence overview (Fateev & Abbas; the design rationale for durable, long-running business logic): uber.com/en/blog/open-source-orchestration-tool-cadence-overview
AWS Step Functions — developer guide (contrast the JSON state-machine / ASL model with code-as-workflow): docs.aws.amazon.com/step-functions/latest/dg/welcome.html
Apache Airflow — core concepts (DAG scheduling + task-level retry, and why it is NOT replay-deterministic): airflow.apache.org/docs/apache-airflow/stable/core-concepts
Azure Durable Functions — concepts (the replay model expressed via generators/yield; a useful third data point): learn.microsoft.com/azure/azure-functions/durable/durable-functions-overview
Stripe — Idempotency keys (the canonical real-world idempotency-key pattern; pair it with the double-charge scene): docs.stripe.com/api/idempotent_requests

More in Consensus, Coordination & Durable Execution

Getting N machines to agree, and getting one job to happen exactly once: Raft, coordination services, CRDTs, locks, leader election, schedulers and durable workflows.

Browse the full problem catalog, or see what the simulator does and does not model.