Metrics / Monitoring System

Time-series at scale. Cardinality is the enemy.

SavedSaved on this device

The problem

Time-series at scale. Cardinality is the enemy.

You are building the monitoring platform for Shopfront, an online store on Kubernetes whose checkout service must never fail silently. At 02:14 on a Tuesday checkout starts returning errors. Either a machine notices within a couple of minutes and pages the on-call engineer, or customers notice first and complain for forty minutes. Everything in this problem exists to make the first outcome true — for one team with two hundred pods, and then for thirty teams with fifty thousand.

build-tsdb built the storage engine that keeps one server's samples. This problem is the product and platform on top of it: how numbers are instrumented and collected, how they are queried without lying, how a query becomes exactly one page, how the whole thing scales past one server's memory, and how it survives the Friday deploy that adds a customer_id label.

The thesis runs through every stage: cardinality is the enemy. A metric's cost ignores traffic and multiplies with every label. Every design decision below is either a way to get more signal per series or a door that stops one team's label explosion from silencing everyone else's alerts. Make each decision yourself, defend it against the workload, and keep one number honest the whole way: time to detect.

What you will cover

metrics vs logs: cost follows label combinations, not traffic
counters, gauges, histograms and cumulative reporting
pull scraping, the up metric, and when push fits
rate() per series before sum(); percentiles from summed buckets
alerting rules with for, routing with grouping and inhibition
meta-monitoring: the always-firing heartbeat and dead man's switch
federation vs remote-write out of one server's memory wall
distributor and ingesters: full-label-set hashing, 3 copies, 2 confirmations
HA pair deduplication: write-time election vs query-time merge
immutable 2 h blocks on object storage and the compactor
query frontend: split by day, shard, cache finished days
churn, relabeling, sample limits, tenant limits, pre-aggregation and exemplars

Reading

Google SRE Book — ch. 6 Monitoring Distributed Systems; SRE Workbook — Alerting on SLOs
Prometheus docs — metric types, histograms and summaries, alerting rules, federation, remote-write tuning
Brian Brazil — Rate then sum, never sum then rate / Cardinality is key (robustperception.io)
Grafana Mimir architecture docs — distributor, ingester, HA deduplication, compactor, query-frontend, limits
Thanos docs — compactor, downsampling and deduplication
Adams et al. — Monarch: Google's Planet-Scale In-Memory Time Series Database (VLDB 2020)
Pelkonen et al. — Gorilla: A Fast, Scalable, In-Memory Time Series Database (VLDB 2015)
AWS Builders' Library — Workload isolation using shuffle-sharding

About Metrics / Monitoring System

Time-series at scale. Cardinality is the enemy.

Difficulty: intermediate
Time: about 108 minutes
Stages: 9
Topic: Observability: Metrics, Logs & Traces

How this problem is worked

Nine stages, from what the thing is for to how it compares with the real implementations. Each asks one question, and the simulator runs the architecture you draw against the requirements you wrote.

01Purpose & invariantsWhat is this for, and what must always be true of it?
02Workload characterizationWho writes, who reads, and in what shapes?
03Data model & on-disk formatWhat does the data look like at rest?
04Core algorithmsHow do the write path and the read path actually work?
05Distribution & replicationHow does this scale out and survive losing a machine?
06Consistency & correctnessUnder concurrency and failure, what is guaranteed?
07Failure modes & recoveryWhat actually happens when each part fails?
08Operational characteristicsCan a human run this at three in the morning?
09Trade-offs & comparisonWhere does this sit against the alternatives?

Primary sources for this problem

Google SRE Book — ch. 6 Monitoring Distributed Systems; SRE Workbook — Alerting on SLOs
Prometheus docs — metric types, histograms and summaries, alerting rules, federation, remote-write tuning
Brian Brazil — Rate then sum, never sum then rate / Cardinality is key (robustperception.io)
Grafana Mimir architecture docs — distributor, ingester, HA deduplication, compactor, query-frontend, limits
Thanos docs — compactor, downsampling and deduplication
Adams et al. — Monarch: Google's Planet-Scale In-Memory Time Series Database (VLDB 2020)
Pelkonen et al. — Gorilla: A Fast, Scalable, In-Memory Time Series Database (VLDB 2015)
AWS Builders' Library — Workload isolation using shuffle-sharding

More in Observability: Metrics, Logs & Traces

The three pillars, built rather than bought: a time-series database, a log pipeline, a tracing system, and the metrics platform on top.

Browse the full problem catalog, or see what the simulator does and does not model.