Membership changes — joint consensus and single-server

Suppose your 3-server cluster needs to become a 5-server cluster: if you let every server flip to the new roster at its own pace, two disjoint majorities can exist long enough to elect two leaders in the same term — breaking Election Safety. The fix is to force every transitional decision through a majority that overlaps BOTH the old and the new rosters; single-server membership change (the production default in etcd, hashicorp/raft, consul) gets that overlap by adding or removing one server at a time, and joint consensus is the paper-correct, more-general alternative.

Previously

Scene 6 closed the safety story for a cluster whose membership is fixed: Election Safety, Leader Append-Only, Log Matching, Leader Completeness, State Machine Safety — five named invariants, one proof. So now the cluster cannot disagree with itself, as long as nobody changes who is IN the cluster. The natural next question is the operational one: how do we change cluster membership — add a server, replace a dead one, migrate to bigger machines — without breaking that same safety chain on the first try?

Scene 07

Membership changes — growing and shrinking the cluster safely

Watch
Try it
Predict
Capture

old roster = {S1..S5}

What to watch for

Suppose your 3-server cluster needs to become a 5-server cluster — or, as drawn here, your 5-server cluster {S1..S5} needs to become {S1, S2, S3, S6, S7}. The obvious plan is to update everyone at once: tell every server about the new roster, restart each one, done. Here's why that breaks. Watch one beat past the restart — that's where the bug lives.

Implementation

Highlighted lines are the ones running in the diagram right now.

Operator.naiveSwap (BAD)

the broken plan: restart each server with the new config at its own pace

1def naiveSwap(oldCluster, newCluster):
2    for server in oldCluster ∪ newCluster:
3        # operator restarts each server independently
4        server.config = newCluster
5        server.restart()
6    # PROBLEM: during this loop, some servers see Cold,
7    # others see Cnew. Two disjoint majorities can form,
8    # both elect for the same term — Election Safety broken.

Leader.addServerSingleStep

production default: learner first, then a single-entry config change

1def addServer(self, newServerId):
2    # 2015 Ongaro fix — gate on current-term commit
3    if not self.hasCommittedEntryInCurrentTerm():
4        return Defer  # wait for no-op-on-election to commit
5    # 1. add as LEARNER: replicates, does not vote
6    self.learners.add(newServerId)
7    self.streamAppendEntriesUntilCaughtUp(newServerId)
8    # 2. propose single-entry config change Cnew = voters ∪ {new}
9    entry = ConfigEntry(
10        kind='add-voter', server=newServerId,
11        Cnew=self.voters ∪ {newServerId})
12    self.appendAndReplicate(entry)
13    # safety: |majority(N)| + |majority(N+1)| > N+1, so any two
14    # such majorities intersect by ≥1 — no disjoint majorities.

Sources

Where this sits in Build Raft — consensus you can defend

Scene 07 of 12. Naive swap creates disjoint majorities and risks Election Safety violation. Single-server changes (production default) preserve majority overlap by N-vs-(N±1); joint consensus C_old,new requires both majorities during transition.

Up next. Membership can change safely. The next operational pressure on the safety chain is log growth: the log cannot be retained forever, so each replica must compact independently by snapshotting its state machine — and that compaction must not break Log Matching either. Scene 8 walks the snapshot mechanism.

All 12 scenes in Build Raft — consensus you can defend · Every curriculum

Built with Arqly

Every scene in Build Raft — consensus you can defend builds on the one before it.All 12 Build Raft — consensus you can defend scenes