Partitions — splitting the log — Build Kafka

Scene 03 · Partitions — splitting the log

Parallelism by sharding ordering. Hot partitions, key skew.

Previously

One log preserves order. But one log is also one machine's throughput. Splitting it into N partitions is how Kafka scales — at the cost of giving up strict GLOBAL order in exchange for per-key order.

Scene 03

Partitions — splitting the log

Watch

Diagram

Producers on the left send keyed records into a topic with N partitions in the middle; the routing arrow shows that hash(key) mod N picks the partition (same key → same lane → ordered). Consumer groups on the right are assigned partitions by the group coordinator — within a group every partition has exactly one owner; across groups every group sees the whole topic.

Sources

Watch each producer's key get hashed into a specific partition. Same key always lands in the same partition — that's how per-key order is preserved. The 2 consumers below belong to the same consumer group; they split the partitions between them.

Implementation

Producer.partitionFor

key → partition, deterministic and stateless

1def partitionFor(record, partitionCount):
2    if record.key is None:
3        # round-robin or sticky batch
4        return next_round_robin(partitionCount)
5    h = murmur2(serialize(record.key))
6    # toPositive masks the sign bit
7    return toPositive(h) % partitionCount

RoundRobinAssignor.assign

split partitions among the members of one group

1def assign(members, partitions):
2    owners = {m: [] for m in members}
3    for j, p in enumerate(partitions):
4        owner = members[j % len(members)]
5        owners[owner].append(p)
6    # members with [] never receive a fetch
7    return owners

Coordinator.onJoinGroup

rerun the assignor whenever group membership changes

1def onJoinGroup(groupId, member):
2    g = groups[groupId]  # per-group state
3    g.members.add(member)
4    g.generationId += 1  # fences old assignments
5    plan = assignor.assign(
6        list(g.members), topic.partitions,
7    )
8    for m, owned in plan.items():
9        m.send(SyncGroup(g.generationId, owned))

Not sure what to ask? Tap a question — the staff engineer answers in the chat panel.

PreviousOffsets, retention, and where bookmarks live NextReplication — ISR is not a quorum