Build Kafka (13 scenes)
Scene 02 · The log is the database
Why a log isn't a queue, and why that one fact unlocks the rest.
Previously
You've seen records flow end-to-end. Now the load-bearing fact: Kafka is NOT a queue. Reads don't pop messages — the consumer is just a bookmark, and the log keeps the bytes.
Scene 02
The log is the database
Diagram
A single partition's log laid out left to right — old records on the left, new on the right, every cell immutable once written. Two consumer groups on the right side each carry their OWN bookmark (offset) into the log; reads do not delete anything, and the two groups can sit at completely different positions.
What Kafka is for: many services need to react to the same stream of events — orders, clicks, signups, metrics — without each one polling a database or calling every consumer directly. Kafka is a durable, append-only log that producers write to and any number of consumers read from independently. The vocabulary in this diagram is the whole thing: a *Producer* sends messages; a *Partition* is the ordered log of those messages (one cell per message, numbered left-to-right by *offset*); a *Consumer* reads from a position on that strip. Watch the producer append messages to the partition. Consumer 1's cursor follows along — it's a *position*, not a queue position. Messages don't disappear when read; the cursor just advances.
Implementation
Log.append
producer-side append to the partition
1def append(record):2 offset = len(log) # next slot3 log.append(record) # write-once cell4 # no overwrite, no in-place mutation5 return offset
Log.read
server-side read at a per-consumer offset
1def read(consumer_id, count = 1):2 pos = cursor[consumer_id]3 records = log[pos : pos + count]4 # slice — log itself is unchanged5 cursor[consumer_id] = pos + len(records)6 return records
Queue.dequeue
the contrast — single reader, destructive read
1def dequeue():2 if not buffer:3 return None4 record = buffer.pop(0) # head removed5 # gone — no second reader can ever see it6 return record
Not sure what to ask? Tap a question — the staff engineer answers in the chat panel.