Design canvas — pick the queue for the workload — Build a Message Queue (RabbitMQ / SQS)

Build a Message Queue (RabbitMQ / SQS) (14 scenes)

Scene 13 · Design canvas — pick the queue for the workload

Capstone: place a workload on the canvas, set every knob, flag whether Kafka would have been a better fit.

Previously

You can read the meters and you can name every mechanism. Last move: stop being told the workload. Pick one, set every knob, and let the verifier challenge you — including the question 'should this have been Kafka?'

Scene 13

Design canvas — pick the queue for the workload

Watch

Diagram

Workload picker on the left (five real workloads), design canvas on the right. Each slot carries a verdict (green/yellow/red) with one-sentence reasoning that cites the earlier scene it references — mq-08 (DLQ threshold), mq-09 (visibility timeout), mq-09a (prefetch), mq-10 (ordering vs parallelism), mq-11 (fanout shape). The bottom cell asks the closing question: would Kafka have been a better fit?

Welcome-email workload is pre-loaded with sensible defaults — 30 s visibility timeout, prefetch=10, maxReceiveCount=5, Standard queue, single pool. Watch the verifier walk each slot in turn and name the earlier scene the default came from. Then switch workloads and watch a default explode.

Implementation

verify(workload, knobs)

runs the verifier pipeline; returns per-slot verdicts

1def verify(workload, knobs):
2    verdicts = {}
3    # mq-09: visibility timeout vs p99 processing.
4    verdicts['visibility'] = check_visibility(workload, knobs)
5    # mq-09a: prefetch vs RTT / processing ratio.
6    verdicts['prefetch']   = check_prefetch(workload, knobs)
7    # mq-08: maxReceiveCount band + transient failures.
8    verdicts['dlq']        = check_max_receive(workload, knobs)
9    # mq-10: queue mode + ordering scope vs need.
10    verdicts['ordering']   = check_ordering(workload, knobs)
11    # mq-11: fanout shape vs consumer-group count.
12    verdicts['fanout']     = check_fanout(workload, knobs)
13    return verdicts

kafka_fit(workload)

the boundary decision tree — queue vs Kafka

1def kafka_fit(workload):
2    # Acks delete (mq-02 / mq-05). No replay surface.
3    if workload.needs_replay:
4        return yes('queue acks delete — no history')
5    # Queue fanout copies the message into N queues (mq-11).
6    if workload.needs_multi_group_on_one_store:
7        return yes('one log, N readers — Kafka costume')
8    # Exactly-once across input+output topics.
9    if workload.needs_eos_input_to_output:
10        return yes('transactional producer + read_committed')
11    # Work-to-do, single pool, idempotent consumer suffices.
12    return no('queue + at-least-once + dedupe is honest')

Broker.process_cell(cell, worker_id)

the load-bearing event — every knob wired in

1def process_cell(cell, worker_id):
2    # mq-09: hide from peers for visibility_timeout_sec.
3    cell.invisible_until = now() + visibility_timeout_sec
4    cell.receive_count += 1
5    # mq-08: too many receives → quarantine, don't redeliver.
6    if cell.receive_count > max_receive_count:
7        dead_letter_queue.send(cell); return
8    # mq-09a: lease counts against worker's prefetch budget.
9    worker.in_flight += 1   # capped at prefetch
10    try:
11        worker.handle(cell.body)
12        ack(cell)           # mq-05: ack means DELETE
13    except TransientError:
14        nack(cell, requeue=True)   # mq-06: back to the pool

PreviousOperations — depth, age, DLQ