Build a distributed search engine (Elasticsearch / OpenSearch style) (12 scenes)
Scene 11 · Operational sharp edges
Refresh storm, mapping explosion, hot shard, deep pagination — the four most expensive Elasticsearch outages are misuses of knobs the earlier scenes already introduced.
Previously
Approximations are mathematical. The next class of failures is operational — same cluster, same algorithms, scale or cardinality has just crept past what one knob's default was set for.
Scene 11
Operational sharp edges
Diagram
Cluster pill at the top reflects the worst-case shard: green = every primary and replica assigned, yellow = some replicas missing (operational), red = some primary unallocated. Six nodes laid out as cards with shard count, heap %, indexing/sec, and search/sec, each tagged with an ILM tier (hot → warm → cold → frozen). On the right, four incident cards — each footer points back at the scene that introduced the underlying mechanism.
Steady state: 3 hot nodes serving live indices, 2 warm nodes holding 30-day-old data, 1 cold node holding searchable snapshots. Status is GREEN — every primary and every replica is assigned. ILM (Index Lifecycle Management) ages indices hot → warm → cold → frozen → deleted on a 90-day schedule.