One server, three ways to die

Build a wide-column store (Cassandra / DynamoDB family) (13 scenes)

Scene 01 · One server, three ways to die

A single box hits a capacity wall, a throughput wall, and a death event — and your data is gone.

Scene 01

Watch

Diagram

One **server** card with three failure widgets: a **disk meter** on the left fills toward 100% as data lands, a **throughput sparkline** along the top saturates a red 'max RPS' ceiling, and a **heartbeat dot** on the right pulses while the process is alive. The **single-node ceiling** is the hard upper bound of capacity and throughput one box can deliver before it fails. **Durability** is the guarantee that an acknowledged write survives a crash — when the heartbeat flatlines and the disk goes dark, that guarantee is gone.

Watch a single box under steady load. The disk meter creeps upward, the throughput sparkline drifts toward the red ceiling. Nothing exotic — just one server, doing its job, getting closer to its limits.

Implementation

Server.put

the single-node write loop and its three failure branches

1def put(key, value):
2    if disk.free_bytes < len(value):
3        raise DiskFull           # capacity wall
4    if cpu.utilization > 1.0 or io.queue_full():
5        raise Saturated          # throughput wall
6    disk.append(key, value)
7    heartbeat.tick()
8    return OK
9 
10# if the process dies between append and fsync,
11# the only copy of the bytes goes with the disk.

HealthCheck.observe

what the watchdog sees from outside the box

1def observe(server):
2    if not server.heartbeat.alive():
3        return DEAD              # data lost with the disk
4    if server.disk.fill_ratio > 0.95:
5        return CAPACITY_PRESSURE
6    if server.throughput_ratio > 0.9:
7        return SATURATED
8    return HEALTHY

NextSplit keys with hash mod N