Hold the write until it wakes up — hinted handoff and the hint TTL

Hinted handoff lets a coordinator absorb a short replica outage by stashing missed writes locally and replaying them when the replica returns — but only within a TTL.

Previously

Partitions are dramatic; the everyday case is a single replica blipping for a few seconds — for that the cluster has a softer trick.

Scene 10

Hold the write until it wakes up

Watch
Try it
Predict
Capture

What to watch for

Replica E just went DOWN. Watch the simulated outage clock tick — for every missed write, the coordinator stashes a hint envelope. The age meter creeps toward the 3h TTL. Within TTL, the envelope replays when E returns; past it, the envelope dissolves.

Implementation

Highlighted lines are the ones running in the diagram right now.

Coordinator.put

the write path with the dead-replica hint branch

1def put(key, value):
2    replicas = ring.replicas_for(key)
3    for r in replicas:
4        if r.alive:
5            send_async(r, Write(key, value))
6        else:
7            # stash on local disk for later replay
8            hint_store.append(
9                Hint(target=r, write=(key, value),
10                     created_at=now()),
11            )
12    return Ack  # to W replicas (best-effort hints)

HintStore.replay_on_return

fires when the down replica gossips back as alive

1def replay_on_return(replica):
2    for hint in hint_store.for(replica):
3        age = now() - hint.created_at
4        if age <= max_hint_window_in_ms:
5            send(replica, hint.write)
6            hint_store.remove(hint)
7        else:
8            # past TTL — replica returns 'stale forever'
9            hint_store.remove(hint)

HintStore.sweep

background TTL sweep — discard hints past max_hint_window

1# runs continuously on the coordinator
2def sweep():
3    for hint in list(hint_store):
4        age = now() - hint.created_at
5        if age > max_hint_window_in_ms:
6            hint_store.remove(hint)  # silent loss
7    sleep(hints_flush_period_ms)

Where this sits in Build a wide-column store (Cassandra / DynamoDB family)

Scene 10 of 13, in the Healing act — Hinted handoff + read repair + anti-entropy + gossip keep the cluster honest.. When a replica is briefly unreachable, the coordinator stashes the write and replays it on return.

Up next. Hints expire and replicas drift cold; the cluster needs a way to find and fix divergence on its own.

All 13 scenes in Build a wide-column store (Cassandra / DynamoDB family) · Every curriculum

Built with Arqly

Every scene in Build a wide-column store (Cassandra / DynamoDB family) builds on the one before it.All 13 Build a wide-column store (Cassandra / DynamoDB family) scenes