Kafka isn't this — reads delete here

A queue's read is destructive — once a consumer takes a message the bytes are gone, the inverse of a Kafka log where a read just advances a per-consumer bookmark and the bytes stay in place.

Previously

We have a buffer between the request and the worker. Same picture as Kafka from a distance, but opposite rules up close: in Kafka the consumer is a bookmark and the bytes stay; here the read removes the cell.

Scene 02

Kafka isn't this — reads delete here

Watch
Try it
Predict
Capture

What to watch for

Same five colored messages, two strips. On the log, both cursors crawl forward and every cell stays lit. On the queue, the first cell vanishes the instant C1 takes it — that's a destructive read.

Implementation

Highlighted lines are the ones running in the diagram right now.

Log.read(consumer_id)

Non-destructive: serve bytes, advance that consumer's cursor.

1def read(consumer_id):
2    offset = cursors[consumer_id]   # per-consumer bookmark
3    if offset >= len(log):
4        return None                 # caught up, bytes stay
5    record = log[offset]            # cell is NOT removed
6    cursors[consumer_id] = offset + 1
7    return record
8 
9def seek(consumer_id, new_offset):
10    cursors[consumer_id] = new_offset  # replay from anywhere

Queue.dequeue()

Destructive: return the head, then remove it. No cursor.

1def dequeue():
2    if head == tail:
3        return None              # queue empty
4    record = cells[head]
5    del cells[head]              # bytes are GONE
6    head += 1                    # broker owns this, not consumer
7    return record
8 
9# No seek(). No replay. A second consumer can only ever
10# receive cells that have not yet been dequeued.

Fanout(message) — same call, opposite shape

Where the bytes live when N consumers want a copy.

1def log_fanout(record):
2    log.append(record)        # one physical cell
3    # each consumer group reads via its own cursor
4 
5def queue_fanout(record, queues):
6    for q in queues:          # one queue per consumer group
7        q.enqueue(copy(record))   # N physical copies
8    # each group dequeues from its own queue

Where this sits in Build a Message Queue (RabbitMQ / SQS)

Scene 02 of 14. Same picture as Kafka from a distance, opposite rules up close: the queue's read removes the cell; no other consumer will ever see it.

Up next. So reads are destructive and the broker tracks state. Time to draw the queue properly — head, tail, and the two operations that move messages across it.

All 14 scenes in Build a Message Queue (RabbitMQ / SQS) · Every curriculum

Built with Arqly

Every scene in Build a Message Queue (RabbitMQ / SQS) builds on the one before it.All 14 Build a Message Queue (RabbitMQ / SQS) scenes