Scene 13

Churn: yesterday's pods still cost you — active series versus series held in memory

Every rolling deploy gives pods new names, and because pod is a label each new name creates brand-new series while the old ones go quiet but stay in memory for hours and in every block index, so cost follows every series seen recently, not the count that is getting samples now.

Previously

Query cost follows the series a query touches; the dashboard's count of series that are getting samples can badly understate that number.

Scene 13

Churn: yesterday's pods still cost you

Watch
Try it
Predict
Capture

What to watch for

Why can a flat count of live series still blow up memory and queries? Checkout runs 10k series per generation of pods, and Shopfront deploys once an hour. Watch the three counters on the right as each deploy lands: the top one, the one every dashboard shows, and the two below it.

Left: one row per generation of checkout pods, newest at the bottom, each row 10k series drawn as tiles.
A green row is getting samples right now; an amber row has gone quiet (its pods are gone) but is still held in memory; a dashed grey row has been written out to a 2 h block and is no longer in memory.
The violet dashed line is the memory cut — the moment head compaction wrote the oldest hours out.
The blue bracket is a query's time range, and it touches every generation it spans.
Right: three counters — active series (series that received a sample recently), series in memory, and series seen today — plus a chip showing churn, the rate at which pod replacement creates brand-new series.

Implementation

Highlighted lines are the ones running in the diagram right now.

Deploy.roll

every replacement gives a pod a brand-new name

1def roll(deployment, every):
2    while True:
3        for pod in deployment.pods:
4            kill(pod)
5            name = f"{deployment}-{rand_suffix()}"
6            pod = start(pod.spec, name = name)
7            # pod and instance are labels
8            pod.labels["pod"] = name
9            pod.labels["instance"] = pod.ip
10        sleep(every)  # 144 rolls/day = every 10 min

Head.append

a label set never seen before becomes a new series

1def append(labels, value, t):
2    ref = series_by_labels.get(hash(labels))
3    if ref is None:
4        ref = head.create_series(labels)
5        head_series_created_total += 1
6    ref.last_sample = t
7    ref.chunk.append(t, value)
8    head_series = len(series_by_labels)
9    active_series = count(last_sample > t - 20m)

Head.compact

the only moment a quiet series leaves memory

1def compact():  # ingesters check every 1 m
2    if head.max_time - head.min_time < 3h:
3        return
4    cut = head.min_time + 2h
5    blocks.write(head.samples_before(cut))
6    head.truncate(before = cut)
7    for ref in series_by_labels.values():
8        if ref.last_sample < cut:
9            series_by_labels.remove(ref)

Query.select

a range walks every block index it overlaps

1def select(matchers, start, end):
2    series = {}
3    for block in blocks.overlapping(start, end):
4        # 2 h blocks: 24 h = 12 of them, 7 d = 84
5        for ref in block.index.matching(matchers):
6            series[ref.labels] = block.chunks(ref)
7    for ref in head.select(matchers, start, end):
8        series[ref.labels] = ref.chunks
9    return evaluate(matchers, series)

Sources

Where this sits in Metrics / Monitoring System

Scene 13 of 18, in the Cardinality act — Churn, defenses, aggregation, then design it.. Every rolling deploy gives pods new names and brand-new series, while the old ones go quiet but linger in memory and in every block index — cost follows series seen recently, not the active count.

Up next. Now that churn and bad labels create series faster than anyone expects, the next question is where an explosion can be stopped, and what each stopping point costs.

All 18 scenes in Metrics / Monitoring System · Every curriculum

Built with Arqly

Every scene in Metrics / Monitoring System builds on the one before it.All 18 Metrics / Monitoring System scenes