Routing — one queue per consumer group

Queue-world fanout is not multiple cursors on one log: it's a router (exchange / SNS topic) that copies each message into one physical queue per consumer group, each with its own backlog, acks, and DLQ.

Previously

Within a queue we partitioned for ordered parallelism. Across queues we need the opposite: one event reaching multiple independent consumer pools, each with its own backlog and retry policy.

Scene 11

Routing — one event, one queue per consumer group

Watch
Try it
Predict
Capture

What to watch for

Watch the producer publish one message at a time. The exchange flashes, then the same payload duplicates into every bound queue at once. Notice that analytics piles up while email and audit drain — each queue has its own backlog.

Implementation

Highlighted lines are the ones running in the diagram right now.

Exchange.publish(message)

for each matching binding, copy into that queue's tail

1def publish(msg, routing_key):
2    for queue, bind_key in bindings[self]:
3        if self.type == 'fanout':
4            queue.tail.append(msg)      # everyone
5        elif self.type == 'direct':
6            if bind_key == routing_key:
7                queue.tail.append(msg)  # exact match
8        elif self.type == 'topic':
9            if pattern_match(bind_key, routing_key):
10                queue.tail.append(msg)  # wildcard
11    # N matches => N physical copies, N backlogs

Exchange.bind(queue, routing_key)

add a binding at runtime; new traffic copied immediately

1def bind(queue, routing_key=''):
2    bindings[self].append((queue, routing_key))
3    # producer code unchanged — no redeploy
4    # queue starts receiving from the NEXT publish
5    # (it does NOT replay history — each queue
6    #  stores its own messages)

ExchangeType.matches

the three matching modes; only one decides delivery

1def matches(exchange_type, bind_key, routing_key):
2    if exchange_type == 'fanout':
3        return True              # ignore key entirely
4    if exchange_type == 'direct':
5        return bind_key == routing_key
6    if exchange_type == 'topic':
7        # bind_key has wildcards: 'order.*'
8        return pattern_match(bind_key, routing_key)

Where this sits in Build a Message Queue (RabbitMQ / SQS)

Scene 11 of 14. Fanout in queue-world is an exchange/SNS topic that copies each message into N physical queues — opposite of Kafka's N cursors on one log.

Up next. We have all the mechanics: ack, nack, DLQ, timeout, prefetch, ordering, routing. In production you don't see those mechanics — you see three dashboards. Which numbers actually wake you up?

All 14 scenes in Build a Message Queue (RabbitMQ / SQS) · Every curriculum

Built with Arqly

Every scene in Build a Message Queue (RabbitMQ / SQS) builds on the one before it.All 14 Build a Message Queue (RabbitMQ / SQS) scenes