Build a Message Queue (RabbitMQ / SQS)

A point-to-point work queue — the messaging primitive Kafka is NOT. Each message goes to one consumer, ack deletes, retries push to a dead-letter queue, and a poisoned message is everyone's problem. Internalize ack vs visibility timeout vs DLQ vs prefetch vs FIFO groups — and learn to tell when Kafka is the wrong tool and when a queue is.

SavedSaved on this device

The problem

A point-to-point work queue — the messaging primitive Kafka is NOT. Each message goes to one consumer, ack deletes, retries push to a dead-letter queue, and a poisoned message is everyone's problem. Internalize ack vs visibility timeout vs DLQ vs prefetch vs FIFO groups — and learn to tell when Kafka is the wrong tool and when a queue is.

You once shoved a job onto SQS or RabbitMQ for an async worker pool and watched it Just Work. Then a worker crashed and the email never sent. Or the visibility timeout fired mid-process and the same payment got applied twice. Or a malformed message hit the queue and the CPU graph went vertical until someone redeployed. Each of those bugs has a name; each name lives in a specific mechanism the broker provides; and each mechanism has two failure modes — one for each end of its knob.

This curriculum is the working developer's guide to those mechanisms, taught in the order the failures show up in production. It is also — explicitly — the contrast curriculum to Kafka. Kafka has its own thirteen scenes in this catalog. This one teaches the other messaging primitive: a queue where each message goes to exactly one consumer, where the consumer's ack is what deletes the message, where retries land in a dead-letter queue, and where you cannot have both global ordering and competing consumers on the same key. By the end you should be able to articulate, sentence by sentence, when Kafka is the wrong tool and when a queue is.

Resist the urge to "describe RabbitMQ" or "describe SQS." Make decisions yourself, defend them, and let the AI push back.

What you will cover

producer / consumer / broker
destructive read vs log read
FIFO strip (enqueue / dequeue)
competing consumers pattern
ack and 'in flight' state
nack, requeue, reject
poison message
dead-letter queue + maxReceiveCount
visibility timeout / ack timeout
prefetch / in-flight cap
FIFO queue + message group
exchange / SNS-fanout routing
queue depth / oldest-message age / DLQ depth

Reading

RabbitMQ Tutorials — Work Queues, Routing, Consumer Acknowledgements (rabbitmq.com/docs/tutorials)
AWS SQS Developer Guide — Standard vs FIFO, Visibility Timeout, Dead-letter queues (docs.aws.amazon.com/AWSSimpleQueueService)
Hohpe & Woolf — Enterprise Integration Patterns: Competing Consumers, Dead Letter Channel, Idempotent Receiver (enterpriseintegrationpatterns.com)
Kleppmann — Designing Data-Intensive Applications, ch. 11 (messaging vs streaming)
AMQP 0-9-1 overview (rabbitmq.com/amqp-0-9-1-overview.html)

About Build a Message Queue (RabbitMQ / SQS)

Difficulty: intermediate
Time: about 80 minutes
Stages: 9
Topic: Queues, Pub/Sub & Event Streaming

How this problem is worked

Nine stages, from what the thing is for to how it compares with the real implementations. Each asks one question, and the simulator runs the architecture you draw against the requirements you wrote.

01Purpose & invariantsWhat is this for, and what must always be true of it?
02Workload characterizationWho writes, who reads, and in what shapes?
03Data model & on-disk formatWhat does the data look like at rest?
04Core algorithmsHow do the write path and the read path actually work?
05Distribution & replicationHow does this scale out and survive losing a machine?
06Consistency & correctnessUnder concurrency and failure, what is guaranteed?
07Failure modes & recoveryWhat actually happens when each part fails?
08Operational characteristicsCan a human run this at three in the morning?
09Trade-offs & comparisonWhere does this sit against the alternatives?

Primary sources for this problem

RabbitMQ Tutorials — Work Queues, Routing, Consumer Acknowledgements (rabbitmq.com/docs/tutorials)
AWS SQS Developer Guide — Standard vs FIFO, Visibility Timeout, Dead-letter queues (docs.aws.amazon.com/AWSSimpleQueueService)
Hohpe & Woolf — Enterprise Integration Patterns: Competing Consumers, Dead Letter Channel, Idempotent Receiver (enterpriseintegrationpatterns.com)
Kleppmann — Designing Data-Intensive Applications, ch. 11 (messaging vs streaming)
AMQP 0-9-1 overview (rabbitmq.com/amqp-0-9-1-overview.html)

More in Queues, Pub/Sub & Event Streaming

Moving events between services without losing them: logs, work queues, fanout, change data capture, stream processing, and the delivery systems built on top.

Browse the full problem catalog, or see what the simulator does and does not model.