How a query becomes points — the four-stage read path, parse to aggregate

A read is four sequential stages — parse selectors, resolve labels to series IDs, decompress the matching chunks for the time range, then aggregate — and only stage 3's cost scales with how far back you look.

Previously

Writes are durable. Now flip the system: a query says {method=POST, status=500} and we need to walk from those labels to actual bytes on disk.

Scene 07

How a query becomes points

Watch
Try it
Predict
Capture

What to watch for

A PromQL query enters on the left. Watch it walk the four stages: parse splits text into selectors and a range, resolve hits a (still-opaque) index that emits a small set of series IDs, decompress unpacks the matching chunks for the [5m] window, and aggregate folds them into one number. The timing bar at the bottom shows where the milliseconds went.

Implementation

Highlighted lines are the ones running in the diagram right now.

TSDB.query

the read path: four sequential stages, one per call

1def query(text, t0, t1, fn):
2    selectors, range = parseQuery(text)
3    series_ids = resolveSeries(selectors)
4    points = decompressAndAggregate(
5        series_ids, t0, t1, fn,
6    )
7    return points

parseQuery

stage 1 — tokenise text into selectors and a range

1def parseQuery(text):
2    ast = promql.parse(text)
3    selectors = []
4    for matcher in ast.label_matchers:
5        selectors.append(
6            (matcher.name, matcher.value),
7        )
8    range = ast.range  # e.g. [5m], [1h], [30d]
9    return selectors, range

resolveSeries

stage 2 — selectors hit the inverted index (black box)

1def resolveSeries(selectors):
2    # postings list per (label, value)
3    # cost depends on cardinality, NOT on range
4    postings = [
5        index.postings(label, value)
6        for (label, value) in selectors
7    ]
8    return intersect(postings)  # → {S3, S7}

decompressAndAggregate

stages 3 & 4 — unpack chunks in [t0,t1], then fold

1def decompressAndAggregate(ids, t0, t1, fn):
2    points = []
3    for sid in ids:
4        for chunk in chunksFor(sid):
5            if chunk.overlaps(t0, t1):
6                # delta-of-delta + XOR decode
7                points += chunk.decompress()
8    return fn(points)  # rate / sum / avg

Where this sits in Build a Prometheus-style time-series database

Scene 07 of 12. A read is four stages — parse, resolve label-selectors to series IDs, decompress the matching chunks, then aggregate. Stage 3 dominates.

Up next. Three of the four stages are clear. Stage 2 is still a black box — how does a database go from {method=POST} to a set of integer series IDs in microseconds?

All 12 scenes in Build a Prometheus-style time-series database · Every curriculum

Built with Arqly

Every scene in Build a Prometheus-style time-series database builds on the one before it.All 12 Build a Prometheus-style time-series database scenes