Build a Prometheus-style time-series database

The simplest database that can absorb a 1M-points-per-second firehose and still answer `sum(rate(http_requests_total{status="500"}[5m]))` in milliseconds — built bit by bit, literally.

SavedSaved on this device

The problem

The simplest database that can absorb a 1M-points-per-second firehose and still answer `sum(rate(http_requests_total{status="500"}[5m]))` in milliseconds — built bit by bit, literally.

You are designing the simplest database that can stomach a metrics firehose: 10,000 hosts each emitting 50 numbers every 15 seconds, forever, without losing precision and without needing a rack of disks. Clients push points in (scrape), or run PromQL-shaped queries pulling them out (sum(rate(metric{labels}[range]))); the engine packs every sample into a few bits on disk and answers a 5-minute alert in single-digit milliseconds.

Prometheus and Gorilla are the canonical worked examples. Once you build the core, the design space of every metrics store collapses to a handful of named trades — the bits-per-point you accept, the cardinality budget you defend, the depth-of-retention you afford, and where you put the line between "single-node" and "somebody else's problem."

Resist the urge to "describe Prometheus." Make decisions yourself, defend them, and let the workload push back. The point is to feel why each price is paid and which workload pays it.

What you will cover

the time-series point shape: (metric, label_set, timestamp, value)
delta-of-delta encoding for fixed-cadence timestamps
XOR encoding for adjacent IEEE-754 floats
chunks: 120 samples or 2 hours of compressed points
head chunk in RAM + write-ahead log + sealed mmapped chunks
inverted index: postings lists per (label, value), intersection at query time
cardinality as the master operational variable
downsampling and the retention pyramid
single-node TSDB + HA via parallel scrapers + remote-write

Reading

Pelkonen et al. — Gorilla: A Fast, Scalable, In-Memory Time Series Database (Facebook, VLDB 2015)
Fabian Reinartz — Writing a Time Series Database from Scratch (fabxc.org/tsdb, 2017)
Ganesh Vernekar — Prometheus TSDB (Part 1): The Head Block
Brian Brazil — Cardinality Is Key (robustperception.io)
Prometheus storage and naming docs (prometheus.io)
InfluxData — New Storage Engine: Time-Structured Merge Tree (TSM)

About Build a Prometheus-style time-series database

The simplest database that can absorb a 1M-points-per-second firehose and still answer `sum(rate(http_requests_total{status="500"}[5m]))` in milliseconds — built bit by bit, literally.

Difficulty: beginner
Time: about 80 minutes
Stages: 9
Topic: Observability: Metrics, Logs & Traces

How this problem is worked

Nine stages, from what the thing is for to how it compares with the real implementations. Each asks one question, and the simulator runs the architecture you draw against the requirements you wrote.

01Purpose & invariantsWhat is this for, and what must always be true of it?
02Workload characterizationWho writes, who reads, and in what shapes?
03Data model & on-disk formatWhat does the data look like at rest?
04Core algorithmsHow do the write path and the read path actually work?
05Distribution & replicationHow does this scale out and survive losing a machine?
06Consistency & correctnessUnder concurrency and failure, what is guaranteed?
07Failure modes & recoveryWhat actually happens when each part fails?
08Operational characteristicsCan a human run this at three in the morning?
09Trade-offs & comparisonWhere does this sit against the alternatives?

Primary sources for this problem

Pelkonen et al. — Gorilla: A Fast, Scalable, In-Memory Time Series Database (Facebook, VLDB 2015)
Fabian Reinartz — Writing a Time Series Database from Scratch (fabxc.org/tsdb, 2017)
Ganesh Vernekar — Prometheus TSDB (Part 1): The Head Block
Brian Brazil — Cardinality Is Key (robustperception.io)
Prometheus storage and naming docs (prometheus.io)
InfluxData — New Storage Engine: Time-Structured Merge Tree (TSM)

More in Observability: Metrics, Logs & Traces

The three pillars, built rather than bought: a time-series database, a log pipeline, a tracing system, and the metrics platform on top.

Browse the full problem catalog, or see what the simulator does and does not model.