All scenes
Build Kafka
13 scenes · ~91 min · build the primitive
Build your own Kafka
A partitioned, replicated, append-only log. The log is the database — internalize that, and a dozen product designs get easier.
- 01Foundations — what Kafka is, words you'll hearWhy Kafka exists and the seven core nouns (producer, broker, topic, partition, record, offset, consumer). Orientation before you touch anything.~7 min
- 01aHello Kafka — topic, brokers, recordsFoundations: what's a topic vs a partition, what's a broker, what does the producer/consumer code actually look like.~7 min
- 02The log is the databaseWhy a log isn't a queue, and why that one fact unlocks the rest.~7 min
- 02aOffsets, retention, and where bookmarks liveRead and commit are separate ack channels; retention, not consumers, ages records out.~7 min
- 03Partitions — splitting the logParallelism by sharding ordering. Hot partitions, key skew.~7 min
- 04Replication — ISR is not a quorumWhy a write commits when the in-sync set fetches it, not a majority.~7 min
- 04aCluster, controller, and metadataOne controller per cluster; KRaft made the metadata itself a Raft log.~7 min
- 05Durability is four knobs, not oneacks, min.insync.replicas, RF, unclean — and how 'all' silently means one.~7 min
- 05aLog compaction — keep the last value per keyCompact-retention turns the log into a state store; tombstones propagate deletes.~7 min
- 06Leader epoch — the vector clock that fixes truncationWhy HW-based truncation could silently lose acked writes, and how KIP-101 closed the gap.~7 min
- 07Rebalance — stop-the-world vs. cooperativeEager revokes everyone; cooperative-sticky only the lanes that move.~7 min
- 08Exactly-once — three monotonic countersPID, epoch, group-generation — three independent fences against zombies.~7 min
- 09Design canvas — pick the knobsCapstone: apply scenes 2-8 to a fresh problem and articulate the trade-off you took.~7 min