Scene 14

Stop the bomb at every door — relabeling, per-scrape sample and per-tenant series limits

Each door stops a cardinality explosion by failing something different: a rule at the scraper drops a dangerous metric before it becomes a series, a per-scrape sample limit fails the whole scrape and silences that target's alerts, and a per-tenant series limit refuses only new series, isolating teams only if each team is its own tenant.

Previously

Churn and bad labels create series faster than anyone expects, so the cluster needs doors that stop an explosion before it takes everyone down.

Scene 14

Stop the bomb at every door

Watch
Try it
Predict
Capture

What to watch for

Where can a cardinality explosion be stopped, and what does each stopping point cost? Start with the answer nobody wants: nowhere. Three teams share one cluster, and the Friday deploy you have met before adds customer_id to checkout's request metric with no door in its way. Watch the rose chip leave checkout's pods, pass the scraper untouched, and turn into series inside the cluster, and watch the memory bar climb toward the budget line while all three lanes still look healthy.

The pipeline strip runs along the top with gate bars on the arrows into collect and into store — those are the doors.
Below it the rose customer_id chip travels from checkout's pods toward whichever door is armed, and the bar on the right is the memory the shared ingesters are using, with a dashed budget line.
The card on the left names the door and its price; the card beside it holds the two rules: relabeling, a scraper rule that drops a whole metric before it can become a series, and the tenant limit, a per-team ceiling on how many series may exist.
The three rows underneath are the teams' alert lanes, each showing whether alerts work, what up reads, and the time-to-detect; the dashed violet box around them is the tenant, either one shared by everyone or one per team.

Implementation

Highlighted lines are the ones running in the diagram right now.

Scraper.relabel

drops a whole metric before it can become a series

1def relabel(samples):      # metric_relabel_configs
2    kept = []
3    for s in samples:
4        # runs as the last step before ingestion
5        if drop_rule_matches(s.name):
6            continue      # never becomes a series
7        kept.append(s)
8    return kept

Scraper.enforce_sample_limit

one scrape, kept whole or thrown away whole

1def enforce_sample_limit(target, samples):
2    # sample_limit default 0 = off
3    if not sample_limit or len(samples) <= sample_limit:
4        commit(samples)
5        write(up = 1)
6        return
7    rollback(pending_append)
8    for series in series_from(target):
9        write_stale_marker(series)
10    write(up = 0)

Ingester.create_series

the ceiling is checked only for a series that does not exist yet

1def create_series(tenant, series, sample):
2    # the ceiling counts every series in this tenant
3    if series in tenant.active_series:
4        append(series, sample)
5        return    # existing series keep ingesting
6    # churn leaves old series in this count too
7    # max_global_series_per_user = 150000
8    if tenant.active_series.count() >= limit:
9        discard(series, 'per_user_series_limit')
10        return
11    tenant.active_series.add(series)
12    append(series, sample)

Sources

Where this sits in Metrics / Monitoring System

Scene 14 of 18, in the Cardinality act — Churn, defenses, aggregation, then design it.. Each door stops a cardinality explosion by failing something different: relabeling drops the metric, a per-scrape limit fails the whole scrape and silences that target, a tenant limit refuses only new series.

Up next. Now that every defense stops the explosion by dropping, blinding or refusing something, the next question is how to keep people's questions answerable while storing far fewer series.

All 18 scenes in Metrics / Monitoring System · Every curriculum

Built with Arqly

Every scene in Metrics / Monitoring System builds on the one before it.All 18 Metrics / Monitoring System scenes