Sentinel — quorum detects, majority elects

Scene 07 · Sentinel — quorum detects, majority elects

SDOWN/ODOWN ladder, epoch-based election, and the operator footgun: quorum ≠ majority.

Previously

Replicas hold a copy, but they don't decide on their own when to take over. Sentinel is the small, separate process that watches, votes, and promotes.

Scene 07

Sentinel — quorum detects, majority elects

Watch

Diagram

Master and replicas in the center, ringed by N Sentinel processes. Each Sentinel pings the master and votes; once `quorum` Sentinels agree it's down (subjective → objective DOWN), an election starts. The election needs a strict MAJORITY of all configured Sentinels — not just `quorum` — to pick a leader and promote a replica; both gauges are shown side by side so you can watch them diverge.

Sources

Three Sentinels watch one master and a replica. The master crashes; watch SDOWN turn into ODOWN once quorum agrees, then a vote in epoch 1 elects the leader who promotes the replica.

Implementation

Sentinel.tick

periodic: detect SDOWN locally, escalate to ODOWN at quorum

1def tick(self):
2    if not master.responds_within(down_after_ms):
3        self.mark_sdown()  # local opinion only
4    # ask peers via SENTINEL is-master-down-by-addr
5    odown_agree_count = 1 + sum(
6        1 for peer in peers if peer.thinks_sdown(master)
7    )
8    if odown_agree_count >= quorum:
9        self.status = ODOWN  # quorum-agreed
10        self.start_election()

Sentinel.startElection

bump epoch, broadcast a vote request, collect peer votes

1def start_election(self):
2    if self.status != ODOWN:
3        return
4    self.my_epoch += 1  # fence this round
5    self.vote_for = self.id
6    broadcast(VoteRequest(
7        candidate = self.id,
8        epoch = self.my_epoch,
9    ))
10    self.tally = collect_votes_from_peers(
11        epoch = self.my_epoch,
12    )

Sentinel.tallyVotes (majority required)

ODOWN used quorum; election uses majority — different counters

1def tally_votes(self):
2    majority = (sentinel_count // 2) + 1  # NOT quorum
3    if max(self.tally.values()) >= majority:
4        winner = argmax(self.tally)
5        promote(replica)  # SLAVEOF NO ONE
6        return
7    if all_voted_and_no_majority(self.tally):
8        self.state = TIED
9        retry_in(2 * failover_timeout)  # bumps epoch

PreviousReplication is async — acked writes can vanish NextCluster — 16384 slots and the client routes