Disks fail weekly — so what does durable mean? — eleven nines and fleet annual failure rate

Run millions of disks at a 1–4% yearly failure rate and a disk dies every few minutes, so 'durable' cannot mean 'the disk doesn't fail' — it has to mean 'the disk failing doesn't lose your bytes', a designed-for target of eleven nines.

Previously

We cracked the bucket open and asked what 'forever' has to survive; the answer starts at the hardware — and the hardware is millions of disks that are dying continuously.

Scene 02

Disks fail weekly — so what does durable mean?

Watch
Try it
Predict
Capture

Eleven nines = the designed-for target: ~1 object lost per 10,000 years per 10 million stored. Essentially never.

What to watch for

Watch the fleet. The clock ticks and disks flip from green to dead-red on a steady beat. One outlined tile holds a tracked object — its only copy. Keep watching that tile.

A grid of disk tiles standing in for the whole fleet (green = alive, red = dead).
A wall clock ticks and disks flip red on a cadence set by the fleet size and the annual failure rate.
One outlined tile is a tracked object that lives on a single disk with no backup; if that disk dies the object is gone.
The header readout names the target — 'eleven nines' means about one object lost per 10,000 years per 10 million stored objects, essentially never — and frames it as a question of whether your bytes still EXIST on some disk (durable), which is a separate promise from whether you can REACH them this instant (available).

Implementation

Highlighted lines are the ones running in the diagram right now.

Fleet.deathCadence

the arithmetic the two sliders drive

1def death_cadence(fleet_size, afr):
2    # afr = per-disk annual failure rate (1%..4%)
3    deaths_per_year = fleet_size * afr
4    seconds_per_year = 365 * 24 * 3600
5    # the gap between two disk deaths, in seconds
6    gap = seconds_per_year / deaths_per_year
7    return gap
8 
9# one disk: gap is months — a death is rare
10# millions of disks: gap collapses to minutes
11# the per-disk rate never changed; the fleet did

Durability.target

what 'eleven nines' names — a target, not a guarantee

1# a designed-for probabilistic target, not a measured SLA
2target = 0.99999999999  # eleven nines / year
3expected_lost_fraction = 1 - target  # 1e-11
4 
5# stored 10_000_000 objects?
6#   expect to lose ~1 object every 10_000 years
7objects = 10_000_000
8years_per_loss = 1 / (objects * expected_lost_fraction)
9 
10# this is DURABLE: do the bytes still EXIST.
11# separate from AVAILABLE: can you REACH them now.

Sources

Where this sits in Build an S3-style distributed object store

Scene 02 of 12, in the The stakes act — What an object store promises — and the durability problem.. Millions of disks at a few % AFR: one dies every few minutes. Eleven nines — and why copies alone are too costly.

Up next. If surviving disk death means keeping the bytes somewhere other than the dead disk, the very first question is: where do those bytes live, and what keeps track of where they are?

All 12 scenes in Build an S3-style distributed object store · Every curriculum

Built with Arqly

Every scene in Build an S3-style distributed object store builds on the one before it.All 12 Build an S3-style distributed object store scenes