Why strong consistency took S3 fourteen years

S3's metadata cache was eventually consistent for 14 years — a write and a read could hit different cache nodes, so a read returned a stale index pointer; the Dec 2020 fix was not to bypass the cache but to make it coherent with a witness that records per-object write order and acts as a read barrier, reloading stale metadata before answering — and it is strong only within one region.

Previously

The bytes are durable, but the index that points at them was, for fourteen years, sometimes lying to readers — and making it stop lying was S3's hardest problem.

Scene 08

Why strong consistency took S3 fourteen years

Watch
Try it
Predict
Capture

What to watch for

Eventual consistency: a PUT updates the index through one cache node, but a GET/LIST served from a DIFFERENT cache node still holds the OLD pointer. Watch three real anomalies replay in turn — each one a reader being told something that isn't true any more.

A client PUTs and GETs through a CACHE LAYER of several nodes sitting in front of the INDEX + PERSISTENCE TIER (the source of truth: key → location).
Under eventual consistency a write updates one cache node while a read reads another — returning the OLD pointer (a stale read).
In strong mode a WITNESS read barrier records per-object write order and reloads any stale cache node before answering.
A cross-region replica lags regardless.

Implementation

Highlighted lines are the ones running in the diagram right now.

IndexCache.read(key)

the read path — eventual returns whatever the node holds

1def read(key):
2    node = cacheRing.nodeFor(key)  # may differ from write's node
3    entry = node.lookup(key)
4    if not cfg.strong:
5        return entry  # may be stale: an older pointer, or a miss
6    # strong: ask the witness whether this entry is current
7    if witness.isStale(key, entry):
8        entry = persistence.load(key)  # reload source of truth
9        node.put(key, entry)           # repair the cache node
10    return entry  # read-your-writes, within this region

Sources

Where this sits in Build an S3-style distributed object store

Scene 08 of 12, in the Make it real act — Strong consistency, multipart upload, lifecycle & tiering.. Eventual-consistency anomalies, the Dec 2020 flip, and the witness read-barrier — strong within a region only.

Up next. With the index now telling the truth, the last piece of the object's life is handling the big uploads that don't fit in a single request — and they come with a famous integrity gotcha.

All 12 scenes in Build an S3-style distributed object store · Every curriculum

Built with Arqly

Every scene in Build an S3-style distributed object store builds on the one before it.All 12 Build an S3-style distributed object store scenes