Build your own workflow engine (Temporal / Airflow / Cadence style)

You have written a function that does a few things in order — charge a card, reserve some inventory, ship a package, send an email. You have written a cron job that retries something that failed. That is your starting point. By the end of this curriculum you will have built — conceptually, layer by layer — a Temporal/Cadence-style durable execution engine, and you will be able to reason about a real workflow: where its state lives, what happens to it when the process crashes mid-run, when a retry is safe, how it waits 30 days without holding a thread, and how to change its code a year after it started — and choose, for a given workload, whether a durable engine, a DAG scheduler, or a state machine is the right tool.

We start from one uncomfortable, concrete fact. Scene 1 runs ORDER #1001: ChargeCard($42) → ReserveInventory(1 Widget) → ShipPackage → SendConfirmationEmail. The process crashes right after the card is charged but before the inventory is reserved. A plain function kept its progress (charged = true) only in RAM, and the crash erased it — so a naive cron retry re-runs from the top and charges the card a second time: $84. That is what "just retry the cron job breaks at the second step" actually looks like. Every later mechanism in the stack exists to plug a worse-placed crash, and we re-ask the same question against each one: did we double-charge, and does ORDER #1001 still finish?

This is a curriculum about a strict density-of-vocabulary discipline: across 13 scenes, each scene introduces at most two new terms, and the diagram carries the load. One running example — ORDER #1001 — is threaded through every scene: you watch its progress get wiped from RAM, then written to a durable event history, then replayed to resume at step 2, then quarantined inside activities, then survive a redeploy, a retry storm, a crash inside the charge, a 30-day wait, a cancel signal, a mid-order failure, an endless billing loop, and finally a code change deployed while the order was still sleeping. The same blue-is-workflow / red-is-activity coloring and the same append-only history strip appear in every scene so the arc reads as one continuous story.

Resist the urge to "describe Temporal." Build each layer because the previous layer's crash forced it, feel the failure mode each one prevents, and let the design canvas at the end push back on whether durable execution was even the right tool.