Gossip keeps everyone informed

Build a wide-column store (Cassandra / DynamoDB family) (13 scenes)

Scene 11.5 · Gossip keeps everyone informed

Each node, every second, swaps state with a few peers; the cluster picture converges without a master.

Previously

Reads, hints, and repairs all need the cluster to know who's alive — that membership picture is itself maintained by a quiet background protocol.

Scene 11.5

Gossip — every node, every second, one random peer

Watch

Diagram

Eight nodes arranged loosely on a circle (deliberately NOT the consistent-hash ring — gossip is a separate protocol that runs alongside it). A green-glow node has heard the new piece of state; a grey node has not. Each round, dashed edges show pairs that just exchanged state — both ends update to the union, so the informed set roughly doubles per round. **Gossip** is an epidemic-style background protocol where each node periodically exchanges state with one random peer; the cluster converges to a consistent membership view in O(log N) rounds. The ROUNDS counter and INFORMED badge at the top track convergence; FULLY CONVERGED appears once every node knows.

Node A has just heard a new piece of state — say, 'node X is down'. Watch one round of gossip at a time: each node picks a random peer and exchanges state. The informed set roughly doubles per round, so 8 nodes converge in about log₂(8) = 3 rounds. No master, no broadcast — just pairwise chatter.

Implementation

Node.gossip_round

one tick — pick a peer, swap state, take the union

1def gossip_round():
2    peer = random.choice(peers - {self})
3    digest = self.state.versions()  # {key: version}
4    delta = peer.exchange(digest)
5    # delta = entries peer has at >= our version
6    self.state.merge(delta)
7    # informed set roughly DOUBLES per round →
8    # cluster converges in O(log N) rounds

Node.exchange

per-key max merge — the convergence engine

1def exchange(peer_digest):
2    delta = {}
3    for key, peer_v in peer_digest.items():
4        if peer_v > self.state[key].version:
5            # peer has fresher; we'll take it on merge
6            pass
7        elif self.state[key].version > peer_v:
8            # we have fresher; ship it back
9            delta[key] = self.state[key]
10    return delta

Node.scheduler

every node runs gossip_round on a fixed period

1# runs on every node, forever, with no master
2loop forever:
3    gossip_round()
4    sleep(gossip_period_ms)  # ~1s in Cassandra

PreviousHeal on read, heal on a schedule NextDesign canvas — pick every knob