Scene 02

The log is the database — per-consumer offsets on an append-only log

Log messages are immutable; consumers are bookmarks tracking their own offset. Reads do not delete.

Previously

You've seen records flow end-to-end. Now the load-bearing fact: Kafka is NOT a queue. Reads don't pop messages — the consumer is just a bookmark, and the log keeps the bytes.

Scene 02

The log is the database

Watch
Try it
Predict
Capture

What to watch for

What Kafka is for: many services need to react to the same stream of events — orders, clicks, signups, metrics — without each one polling a database or calling every consumer directly. Kafka is a durable, append-only log that producers write to and any number of consumers read from independently. The vocabulary in this diagram is the whole thing: a Producer sends messages; a Partition is the ordered log of those messages (one cell per message, numbered left-to-right by offset); a Consumer reads from a position on that strip. Watch the producer append messages to the partition. Consumer 1's cursor follows along — it's a position, not a queue position. Messages don't disappear when read; the cursor just advances.

A single partition's log laid out left to right — old records on the left, new on the right, every cell immutable once written.
Two consumer groups on the right side each carry their OWN bookmark (offset) into the log; reads do not delete anything, and the two groups can sit at completely different positions.

Implementation

Highlighted lines are the ones running in the diagram right now.

Log.append

producer-side append to the partition

1def append(record):
2    offset = len(log)         # next slot
3    log.append(record)        # write-once cell
4    # no overwrite, no in-place mutation
5    return offset

Log.read

server-side read at a per-consumer offset

1def read(consumer_id, count = 1):
2    pos = cursor[consumer_id]
3    records = log[pos : pos + count]
4    # slice — log itself is unchanged
5    cursor[consumer_id] = pos + len(records)
6    return records

Queue.dequeue

the contrast — single reader, destructive read

1def dequeue():
2    if not buffer:
3        return None
4    record = buffer.pop(0)    # head removed
5    # gone — no second reader can ever see it
6    return record

Sources

Where this sits in Build Kafka

Scene 02 of 13, in the Why a log? act — Orientation — the log is the database, not a queue.. Why a log isn't a queue, and why that one fact unlocks the rest.

Up next. Offsets, retention, and where bookmarks live — if reads don't delete, what does? And where does a consumer's bookmark physically live?

All 13 scenes in Build Kafka · Every curriculum

Designs that use this

Built with Arqly

Every scene in Build Kafka builds on the one before it.All 13 Build Kafka scenes