Scene 15

Keep the answer, drop the series — pre-aggregation on arrival and bucket exemplars

Summing away the labels nobody queries as samples arrive cuts stored series by orders of magnitude, while deleting every distinguishing label makes counters collide and rates lie. One saved example request keeps the path to a customer without a series per customer.

Previously

Every defense stops the explosion by dropping, blinding or refusing something; we want the answers without the series.

Scene 15

Keep the answer, drop the series

Watch
Try it
Predict
Capture

every dashboard panel is drawn from these 50 series

What to watch for

How do you keep people's questions answerable while storing far fewer series? Start by looking at the gap. Checkout's dashboards only ever query by route and status — 50 series in total. Watch the labels the scrape actually carries pile up beside them, and watch the stored bar leave the queried bar behind.

Left: the labels arriving on checkout's request counter — route, status, pod, instance, and the Friday deploy's rose customer_id chip.
Middle: the label strategy applied to them — keep them, delete them at the scraper, or sum them away as the samples arrive (pre-aggregation).
Right: how many series get stored versus how many the dashboards actually query (log scale), and checkout's request-rate line, with rejection chips when samples collide.
Below: checkout's latency buckets; a pin on a bucket is an exemplar — one example request's trace id saved beside the count.
Top-right is time-to-detect.

Implementation

Highlighted lines are the ones running in the diagram right now.

Scraper.drop_labels

deleting a label deletes what tells two writers apart

1def drop_labels(samples, drop):     # labeldrop
2    if not drop:
3        return samples              # stored as scraped
4    for s in samples:
5        for name in drop:           # ["pod", "instance"]
6            s.labels.pop(name)
7        s.key = series_key(s.labels)   # route, status
8    return samples                  # 50 pods -> one key

Ingester.append

every sample for one key is written on one timeline

1def append(key, t, value):
2    last = series[key].last_sample
3    if last and t == last.t:
4        return reject("duplicate sample")
5    if last and t < last.t:
6        return reject("out of order")
7    series[key].push(t, value)

Aggregator.on_sample

sums the label away as samples arrive, before storage

1DROP = ["pod", "instance"]     # stream aggregation
2 
3def on_sample(s):
4    key = series_key(without(s.labels, DROP))
5    src = (key, s.labels["pod"])   # one state per pod
6    delta = max(0, s.value - last.get(src, s.value))
7    last[src] = s.value
8    total[key] += delta
9 
10def flush():                       # every interval
11    for key, v in total.items():
12        remote_write(key, v)       # only the rollup

App.observe

pins one example request beside the bucket it landed in

1def observe(seconds, trace_id):
2    for b in buckets:              # 0.1, 0.5, 1, 5, +Inf
3        if seconds <= b.le:
4            b.count += 1
5            b.exemplar = Exemplar(trace_id, seconds)
6            break
7 
8def expose(b):                     # OpenMetrics text
9    line = f"{b.count}"
10    if b.exemplar:                 # one per bucket
11        line += f' # {{trace_id="{b.exemplar.id}"}}'
12    return line                    # labels <= 128 chars

Sources

Where this sits in Metrics / Monitoring System

Scene 15 of 18, in the Cardinality act — Churn, defenses, aggregation, then design it.. Summing away labels nobody queries as samples arrive cuts series by orders of magnitude, while deleting every distinguishing label makes counters collide — and an exemplar keeps one example customer.

Up next. Now that you can control what every series costs without losing the answers, the next question is which combination of all these choices a real workload actually needs.

All 18 scenes in Metrics / Monitoring System · Every curriculum

Built with Arqly

Every scene in Metrics / Monitoring System builds on the one before it.All 18 Metrics / Monitoring System scenes