Scene 03

Partitions — splitting the log

A partition is the unit of parallelism. Same key → same partition (per-key order preserved); ordering is sharded across partitions. Within a group, M > N consumers leaves some idle; N > M means some consumers own multiple. Picking partition count is picking your maximum future parallelism — rebalancing later is expensive.

Previously

One log preserves order. But one log is also one machine's throughput. Splitting it into N partitions is how Kafka scales — at the cost of giving up strict GLOBAL order in exchange for per-key order.

Scene 03

Partitions — splitting the log

Watch
Try it
Predict
Capture

What to watch for

Watch each producer's key get hashed into a specific partition. Same key always lands in the same partition — that's how per-key order is preserved. The 2 consumers below belong to the same consumer group; they split the partitions between them.

Producers on the left send keyed records into a topic with N partitions in the middle; the routing arrow shows that hash(key) mod N picks the partition (same key → same lane → ordered).
Consumer groups on the right are assigned partitions by the group coordinator — within a group every partition has exactly one owner; across groups every group sees the whole topic.

Implementation

Highlighted lines are the ones running in the diagram right now.

Producer.partitionFor

key → partition, deterministic and stateless

1def partitionFor(record, partitionCount):
2    if record.key is None:
3        # round-robin or sticky batch
4        return next_round_robin(partitionCount)
5    h = murmur2(serialize(record.key))
6    # toPositive masks the sign bit
7    return toPositive(h) % partitionCount

RoundRobinAssignor.assign

split partitions among the members of one group

1def assign(members, partitions):
2    owners = {m: [] for m in members}
3    for j, p in enumerate(partitions):
4        owner = members[j % len(members)]
5        owners[owner].append(p)
6    # members with [] never receive a fetch
7    return owners

Coordinator.onJoinGroup

rerun the assignor whenever group membership changes

1def onJoinGroup(groupId, member):
2    g = groups[groupId]  # per-group state
3    g.members.add(member)
4    g.generationId += 1  # fences old assignments
5    plan = assignor.assign(
6        list(g.members), topic.partitions,
7    )
8    for m, owned in plan.items():
9        m.send(SyncGroup(g.generationId, owned))

Sources

Where this sits in Build Kafka

Scene 03 of 13, in the Write side act — Partitioning, replication, and durability knobs.. Parallelism by sharding ordering. Hot partitions, key skew.

Up next. Replication — a partition lives on more than one broker. ISR (in-sync replicas) is what decides when a write is safely committed, and it's NOT a quorum vote.

All 13 scenes in Build Kafka · Every curriculum

Designs that use this

Built with Arqly

Every scene in Build Kafka builds on the one before it.All 13 Build Kafka scenes