Scene 07

Rebalance — stop-the-world vs. cooperative

Eager rebalance revokes every partition from every consumer on any membership change, while cooperative rebalance (KIP-429) only revokes partitions that are actually moving — turning a 14-minute outage into a 1-minute brief gap on a couple of lanes.

Previously

Followers and leaders are sorted. Now the OTHER source of churn: consumers come and go, and the group has to reassign partitions. Eager rebalance is a 14-minute outage; cooperative is a 1-minute brief gap.

Scene 07

Rebalance — stop-the-world vs. cooperative

Watch
Try it
Predict
Capture

What to watch for

Four consumers split six partitions. Watch the lanes — they're all green (active). At tick 8, C0 exceeds max.poll.interval.ms; the group enters PreparingRebalance and every lane goes dark for a few ticks. Watch what happens to the lanes that AREN'T moving.

A consumer group with several consumers and a topic with multiple partitions.
When a consumer joins or leaves, the group coordinator triggers a rebalance: 'eager' protocol revokes EVERY partition from EVERY consumer (red strip across all lanes) before reassigning; 'cooperative-sticky' (KIP-429) only revokes the partitions that actually need to move.
The per-consumer timeline strips show the difference — under eager the dark idle band spans every lane; under cooperative-sticky it's a brief gap on only the moving lanes.

Implementation

Highlighted lines are the ones running in the diagram right now.

Coordinator.onJoinGroup # eager

every member revokes everything before reassign

1def onJoinGroup(member):
2    group.state = PreparingRebalance
3    # signal EVERY member to drop EVERY partition
4    for m in group.members:
5        m.send(RevokeAll)
6        await m.onPartitionsRevoked_done
7    # group is idle here — nobody owns anything
8    plan = assignor.assign(group.members,
9                           group.subscribed_topics)
10    group.generation += 1
11    for m, parts in plan.items():
12        m.send(SyncGroup(parts))
13    group.state = Stable

Consumer.onPartitionsRevoked

user callback — why every revoke costs wall-clock

1def onPartitionsRevoked(partitions):
2    # everything below runs while the lane is BLACK
3    for p in partitions:
4        consumer.commitSync(offsets[p])
5        stateStore[p].flush()
6        stateStore[p].close()
7    # Streams: local RocksDB rebuilt on the next assignment
8    metrics.record('revoke.latency', now() - t0)
9    # only after this returns does Coordinator proceed

Sources

Where this sits in Build Kafka

Scene 07 of 13, in the Scale act — Rebalancing without halting every consumer.. Eager revokes everyone; cooperative-sticky only the lanes that move.

Up next. Exactly-once — when retries, restarts, and rebalances all stack up, what stops a duplicate or a zombie writer? Three independent monotonic counters do it.

All 13 scenes in Build Kafka · Every curriculum

Designs that use this

Built with Arqly

Every scene in Build Kafka builds on the one before it.All 13 Build Kafka scenes