Arqly
Sign in
Build a Message Queue (RabbitMQ / SQS)
14 scenes · ~98 min · build the primitive

Build your own Message Queue (RabbitMQ / SQS)

A point-to-point work queue — the messaging primitive Kafka is NOT. Each message goes to one consumer, ack deletes, retries push to a dead-letter queue, and a poisoned message is everyone's problem. Internalize ack vs visibility timeout vs DLQ vs prefetch vs FIFO groups — and learn to tell when Kafka is the wrong tool and when a queue is.

  1. 01
  2. 02
  3. 03
  4. 04
  5. 05
  6. 06
  7. 07
  8. 08
  9. 09
  10. 09a
  11. 10
  12. 11
  13. 12
  14. 13
  1. 01
    The work that can't block the request
    Some side effects (email, PDF, transcode) are too slow for the HTTP request — drop them on a buffer and let a worker pool finish them asynchronously.
    ~7 min
  2. 02
    Kafka isn't this — reads delete here
    Same picture as Kafka from a distance, opposite rules up close: the queue's read removes the cell; no other consumer will ever see it.
    ~7 min
  3. 03
    The queue — head, tail, enqueue, dequeue
    Tail on the right where producers push, head on the left where consumers pull, and the FIFO order between them.
    ~7 min
  4. 04
    Competing consumers — N workers, one head
    Point N workers at the same head; the broker hands each cell to whichever worker is ready. Throughput scales linearly until producer rate caps it.
    ~7 min
  5. 05
    Ack — the consumer's 'you may delete'
    Dequeue is two steps: lease (hide from peers) + ack (delete). Same word as Kafka's ack, but consumer→broker direction — and auto-ack loses messages.
    ~7 min
  6. 06
    Nack and requeue — the failure verdict
    Three consumer verdicts (ack / nack-with-requeue / reject-without-requeue) — and requeue-forever is a footgun.
    ~7 min
  7. 07
    The poison message — the loop that won't end
    A naive requeue loop turns one un-processable message into an infinite redelivery storm that pegs CPU and starves the queue behind it.
    ~7 min
  8. 08
    Dead-letter queue — the escape valve
    After N redeliveries, the broker routes the cell to a sibling DLQ; the main pool keeps moving. maxReceiveCount is the knob with two failure modes.
    ~7 min
  9. 09
    Visibility timeout — the silent-worker clock
    Per-message countdown; if no ack arrives, the broker redelivers. Too short → duplicates; too long → stuck pool; heartbeat extend is the fix.
    ~7 min
  10. 09a
    Prefetch — how many in flight per worker
    Per-worker cap on un-acked in-flight messages. Too low → throughput collapse on RTT; too high → one greedy worker hoards.
    ~7 min
  11. 10
    Ordering vs parallelism — partition the keyspace
    Globally ordered + competing consumers is impossible. FIFO preserves order WITHIN a message group; many groups = parallelism. Kafka's partition-by-key in queue costume.
    ~7 min
  12. 11
    Routing — one queue per consumer group
    Fanout in queue-world is an exchange/SNS topic that copies each message into N physical queues — opposite of Kafka's N cursors on one log.
    ~7 min
  13. 12
    Operations — depth, age, DLQ
    Three numbers tell you a queue's health: visible depth, oldest-message age, DLQ depth. Each one points at a distinct underlying failure.
    ~7 min
  14. 13
    Design canvas — pick the queue for the workload
    Capstone: place a workload on the canvas, set every knob, flag whether Kafka would have been a better fit.
    ~7 min