Build your own Message Queue (RabbitMQ / SQS)
A point-to-point work queue — the messaging primitive Kafka is NOT. Each message goes to one consumer, ack deletes, retries push to a dead-letter queue, and a poisoned message is everyone's problem. Internalize ack vs visibility timeout vs DLQ vs prefetch vs FIFO groups — and learn to tell when Kafka is the wrong tool and when a queue is.
- Scenes
- 14 interactive scenes
- Time
- about 98 minutes
- Topic
- Queues, Pub/Sub & Event Streaming
What you are building, and why
You once shoved a job onto SQS or RabbitMQ for an async worker pool and watched it Just Work. Then a worker crashed and the email never sent. Or the visibility timeout fired mid-process and the same payment got applied twice. Or a malformed message hit the queue and the CPU graph went vertical until someone redeployed. Each of those bugs has a name; each name lives in a specific mechanism the broker provides; and each mechanism has two failure modes — one for each end of its knob.
This curriculum is the working developer's guide to those mechanisms, taught in the order the failures show up in production. It is also — explicitly — the contrast curriculum to Kafka. Kafka has its own thirteen scenes in this catalog. This one teaches the other messaging primitive: a queue where each message goes to exactly one consumer, where the consumer's ack is what deletes the message, where retries land in a dead-letter queue, and where you cannot have both global ordering and competing consumers on the same key. By the end you should be able to articulate, sentence by sentence, when Kafka is the wrong tool and when a queue is.
Resist the urge to "describe RabbitMQ" or "describe SQS." Make decisions yourself, defend them, and let the AI push back.
What you will be able to explain afterwards
- producer / consumer / broker
- destructive read vs log read
- FIFO strip (enqueue / dequeue)
- competing consumers pattern
- ack and 'in flight' state
- nack, requeue, reject
- poison message
- dead-letter queue + maxReceiveCount
- visibility timeout / ack timeout
- prefetch / in-flight cap
- FIFO queue + message group
- exchange / SNS-fanout routing
- queue depth / oldest-message age / DLQ depth
- 01The work that can't block the request — buffering work for an async worker poolSome side effects (email, PDF, transcode) are too slow for the HTTP request — drop them on a buffer and let a worker pool finish them asynchronously.~7 min
- 02Kafka isn't this — reads delete hereSame picture as Kafka from a distance, opposite rules up close: the queue's read removes the cell; no other consumer will ever see it.~7 min
- 03The queue — head, tail, enqueue, dequeueTail on the right where producers push, head on the left where consumers pull, and the FIFO order between them.~7 min
- 04Competing consumers — N workers, one headPoint N workers at the same head; the broker hands each cell to whichever worker is ready. Throughput scales linearly until producer rate caps it.~7 min
- 05Ack — the consumer's 'you may delete' — the lease-and-ack dequeue handshakeDequeue is two steps: lease (hide from peers) + ack (delete). Same word as Kafka's ack, but consumer→broker direction — and auto-ack loses messages.~7 min
- 06Nack and requeue — the failure verdictThree consumer verdicts (ack / nack-with-requeue / reject-without-requeue) — and requeue-forever is a footgun.~7 min
- 07The poison message — the loop that won't endA naive requeue loop turns one un-processable message into an infinite redelivery storm that pegs CPU and starves the queue behind it.~7 min
- 08Dead-letter queue — the escape valveAfter N redeliveries, the broker routes the cell to a sibling DLQ; the main pool keeps moving. maxReceiveCount is the knob with two failure modes.~7 min
- 09Visibility timeout — the silent-worker clockPer-message countdown; if no ack arrives, the broker redelivers. Too short → duplicates; too long → stuck pool; heartbeat extend is the fix.~7 min
- 09aPrefetch — how many in flight per workerPer-worker cap on un-acked in-flight messages. Too low → throughput collapse on RTT; too high → one greedy worker hoards.~7 min
- 10Ordering vs parallelism — partition the keyspaceGlobally ordered + competing consumers is impossible. FIFO preserves order WITHIN a message group; many groups = parallelism. Kafka's partition-by-key in queue costume.~7 min
- 11Routing — one queue per consumer groupFanout in queue-world is an exchange/SNS topic that copies each message into N physical queues — opposite of Kafka's N cursors on one log.~7 min
- 12Operations — depth, age, DLQThree numbers tell you a queue's health: visible depth, oldest-message age, DLQ depth. Each one points at a distinct underlying failure.~7 min
- 13Design canvas — pick the queue for the workloadCapstone: place a workload on the canvas, set every knob, flag whether Kafka would have been a better fit.~7 min
More in Queues, Pub/Sub & Event Streaming
Moving events between services without losing them: logs, work queues, fanout, change data capture, stream processing, and the delivery systems built on top.
- Build Build KafkaA partitioned, replicated, append-only log. The log is the database — internalize that, and a dozen product designs get easier.
- Build Build a CDC pipeline (Debezium + outbox)Your service writes to its DB and publishes to Kafka — and any crash between those two writes is permanent inconsistency. Build a Change Data Capture pipeline (modeled on Debezium + the outbox pattern) that closes the gap by making the database itself the event source.
- Ad Click AggregatorStream processing with watermarks and exactly-once.
- Notification SystemPush, email, SMS. Idempotent. Failover.
- Distributed Cron — Mass Scheduled EmailSingle trigger, 50M recipient idempotency, catch-up.
Prefer to design it yourself?
The same subject as a staged workspace: draw the architecture, and a simulator traces requests through the boxes you drew.
Open the Build a Message Queue (RabbitMQ / SQS) workspace