Scene 02

Cost grows with labels, not traffic — label cardinality and the series count

The app reports one number per interval for each distinct combination of labels, so a metric's cost ignores traffic but multiplies with every label, and one label with unlimited values like customer_id costs more than a 100× traffic spike.

Previously

A rule needs numbers to check; before collecting them everywhere, we need to know what one of those numbers costs to keep.

Scene 02

Cost grows with labels, not traffic

Watch
Try it
Predict
Capture

What to watch for

Why do metrics stay cheap as traffic grows, and what makes them expensive? Watch checkout's traffic climb from 10 to 10,000 requests per second: the log bytes climb with it, and the series count for checkout's request metric (a series is short for time series, one number reported every interval) stays put. Then the on-call engineer needs to know which route is failing, so the number gets tagged, and the tree in the middle starts to multiply.

Left: checkout's traffic gauge feeding two bars, log bytes (one line per request) and series count (how many numbers the app reports each interval).
Center: a multiplication tree, service × route × status × pod × customer_id, each level showing how many values it has and the running total of series; the rose customer_id chip is the Payments team's Friday deploy.
Right: monitoring memory against a dashed budget line (an example budget, not a real limit), at a few KiB per series including overhead (measured by Robust Perception at about 3 kB).
A label is a name=value tag on a metric, like route=/pay, that splits one number into one per value.
Cardinality is how many distinct values a label can take; multiplied across labels, it sets the series count.
Top-right: time-to-detect (example).

Implementation

Highlighted lines are the ones running in the diagram right now.

App.handle_request

every request writes a log line and bumps a count

1def handle_request(req):
2    resp = checkout(req)
3    log.write(req, resp)
4    labels = {
5        "service": "checkout",
6        "route": req.route,
7        "status": resp.status,
8        "pod": POD_NAME,
9        "customer_id": req.customer_id,
10    }
11    request_count[labels] += 1

App.report

one value per label combination, per interval

1every interval:  # e.g. every 15 s
2    for labels, value in request_count.items():
3        send(labels, value, now())

Monitor.store

memory is held per series, not per request

1def store(labels, value, ts):
2    series = active.get(labels)
3    if series is None:
4        series = new_series(labels)  # ~3 kB (measured)
5        active[labels] = series
6    series.append(ts, value)
7 
8def check_memory():
9    used = len(active) * SERIES_BYTES
10    if used > memory_budget:
11        crash("out of memory")

Sources

Where this sits in Metrics / Monitoring System

Scene 02 of 18, in the Why measure act — Numbers beat complaints; labels, not traffic, set the bill.. The app reports one number per label combination per interval, so a metric's cost ignores traffic and multiplies with every label: one unbounded label like customer_id dwarfs a 100× traffic spike.

Up next. Now that each series is one number the app reports every interval, the next question is what that number should mean, a total so far or a count since the last read, so a missed or repeated read doesn't corrupt it.

All 18 scenes in Metrics / Monitoring System · Every curriculum

Built with Arqly

Every scene in Metrics / Monitoring System builds on the one before it.All 18 Metrics / Monitoring System scenes