Scene 02a

Offsets, retention, and where bookmarks live

A consumer's offset is just another piece of data stored in the __consumer_offsets topic; reading and committing are separate ack channels, and retention — not the consumer — decides when records age out.

Previously

Reads don't delete and consumers carry their own bookmark. So who DOES delete records, and where does that bookmark physically live? Both answers are themselves Kafka topics.

Scene 02a

Offsets, retention, and where bookmarks live

Watch
Try it
Predict
Capture

What to watch for

Watch the read and committed offsets — they start in lockstep. They don't have to. Retention crawls in from the left; messages stay until retention ages them out.

The same partition log, with two extra layers visible: a separate __consumer_offsets topic where each consumer's commit position is itself written as a record, and a retention bar at the bottom that ages out the oldest cells on a clock or size budget.
Read and commit are drawn as two independent arrows from each consumer.

Implementation

Highlighted lines are the ones running in the diagram right now.

Consumer.poll

advances the in-memory cursor; auto-commit piggybacks here

1def poll():
2    resp = broker.fetch(
3        topic, partition,
4        fromOffset = self.read,
5    )
6    for record in resp.records:
7        self.read += 1   # in-memory only
8        yield record
9    if enable.auto.commit:
10        # every auto.commit.interval.ms
11        commitOffset(self.read)

Consumer.commitOffset

persists the bookmark — a write into __consumer_offsets

1def commitOffset(offset):
2    # __consumer_offsets is a normal Kafka topic;
3    # the key is (group, topic, partition).
4    record = OffsetCommit(
5        group   = self.group,
6        topic   = topic,
7        part    = partition,
8        offset  = offset,
9    )
10    coordinator.append(record)
11    self.committed = offset

LogCleaner.maybeRoll

broker-side retention — ages segments out by time or size

1# runs continuously per partition
2def maybeRoll():
3    for seg in segments:
4        tooOld   = age(seg) > log.retention.ms
5        tooLarge = totalBytes() > log.retention.bytes
6        if tooOld or tooLarge:
7            delete(seg)   # baseOffset gone
8    if active.size >= log.segment.bytes:
9        roll()           # start a new segment

Sources

Where this sits in Build Kafka

Scene 02a of 13, in the Why a log? act — Orientation — the log is the database, not a queue.. Read and commit are separate ack channels; retention, not consumers, ages records out.

Up next. Partitions — splitting the log. One partition is one ordering lane; multiple partitions trade strict global order for parallelism, and the key picks the lane.

All 13 scenes in Build Kafka · Every curriculum

Designs that use this

Built with Arqly

Every scene in Build Kafka builds on the one before it.All 13 Build Kafka scenes