One server, three ways to die — the single-node ceiling on capacity and throughput

A single server has three failure modes — disk full, throughput saturated, node dead — and any one of them ends your service.

Scene 01

One server, three ways to die

Watch
Try it
Predict
Capture

What to watch for

Watch a single box under steady load. The disk meter creeps upward, the throughput sparkline drifts toward the red ceiling. Nothing exotic — just one server, doing its job, getting closer to its limits.

Implementation

Highlighted lines are the ones running in the diagram right now.

Server.put

the single-node write loop and its three failure branches

1def put(key, value):
2    if disk.free_bytes < len(value):
3        raise DiskFull           # capacity wall
4    if cpu.utilization > 1.0 or io.queue_full():
5        raise Saturated          # throughput wall
6    disk.append(key, value)
7    heartbeat.tick()
8    return OK
9 
10# if the process dies between append and fsync,
11# the only copy of the bytes goes with the disk.

HealthCheck.observe

what the watchdog sees from outside the box

1def observe(server):
2    if not server.heartbeat.alive():
3        return DEAD              # data lost with the disk
4    if server.disk.fill_ratio > 0.95:
5        return CAPACITY_PRESSURE
6    if server.throughput_ratio > 0.9:
7        return SATURATED
8    return HEALTHY

Where this sits in Build a wide-column store (Cassandra / DynamoDB family)

Scene 01 of 13, in the Why distribute? act — One server has three ways to die — and bigger boxes never fix all three.. A single box hits a capacity wall, a throughput wall, and a death event — and your data is gone.

Up next. If one box has three ways to die, the obvious move is more boxes — so the very next question is how to split keys across them.

All 13 scenes in Build a wide-column store (Cassandra / DynamoDB family) · Every curriculum

Built with Arqly

Every scene in Build a wide-column store (Cassandra / DynamoDB family) builds on the one before it.All 13 Build a wide-column store (Cassandra / DynamoDB family) scenes