Design canvas: defend your durability number

Build an S3-style distributed object store (12 scenes)

Scene 11 · Design canvas: defend your durability number

Assemble code, placement, consistency, and lifecycle for a workload — then survive a correlated-failure objection.

Previously

We've built every piece — keyspace, immutability, erasure coding, placement, the two planes, repair, consistency, multipart, lifecycle — so the last move is to assemble them for a real workload and defend the durability number.

Scene 11

Design canvas: defend your durability number

Watch

Diagram

Left: a KNOB palette — the design decisions you ship (coding scheme, small-object threshold, placement across failure domains, consistency story, lifecycle, versioning). Center: a WORKLOAD card with the salient params that drive those knobs. Right: a VERIFIER that grades each knob and cites the earlier scene whose rule it satisfies (s3-04 replicate-small/EC-large, s3-04a placement across failure domains, s3-07 strong-in-region, s3-06 scrub-and-repair, s3-09 lifecycle). Bottom: the correlated-failure objection — kill a rack against the chosen placement — plus a projected-durability readout. Spread placement survives the objection; co-located placement turns the nines into a paper number.

Sources

Workload A — media/backups. The defensible config is pre-loaded: erasure-code the large blobs, spread fragments across failure domains, strong consistency in-region with async cross-region replication, lifecycle to archive. The verifier walks green and every check names the earlier scene it satisfies; the durability readout shows the projected nines.

Implementation

Verifier.gradeConfig

grade each knob against the workload; cite the scene

1def gradeConfig(workload, knobs):
2    for obj_size in workload.object_sizes:
3        # replicate small, erasure-code large  (s3-04)
4        if obj_size < knobs.small_object_threshold:
5            require(knobs.scheme == REPLICATE)
6        else:
7            require(knobs.scheme == ERASURE_CODE)
8    # EC on tiny objects amplifies bytes 2.8-3.3x
9    if workload.dominant_size < 1_KB:
10        reject_if(knobs.scheme == ERASURE_CODE, cite='s3-04')
11    require(knobs.placement == SPREAD, cite='s3-04a')

Verifier.killRack

the correlated-failure objection against the placement

1def killRack(fragments, k, m, placement):
2    # group the k+m fragments by failure domain
3    per_domain = group_by(fragments, lambda f: f.domain)
4    worst = max(len(g) for g in per_domain.values())
5    # one domain dies -> all its fragments vanish at once
6    if worst > m:
7        return DATA_LOST   # exceeds the code's tolerance
8    # <= m lost: repair reads k survivors, rebuilds (s3-06)
9    return SURVIVES

Durability.projectNines

the rate equation behind the eleven-nines readout

1def projectNines(failure_rate, mttr, m, placement):
2    # the model ASSUMES failures are independent
3    if placement != SPREAD:
4        return INVALID  # co-location voids the premise
5    # P(loss) ~ (failure_rate * mttr) ^ (m+1)
6    p_loss = (failure_rate * mttr) ** (m + 1)
7    return nines_of(1 - p_loss)  # repair (mttr) sets the window

Not sure what to ask? Tap a question — the staff engineer answers in the chat panel.

PreviousLifecycle and tiering: cheap because bytes are immutable