Scene 05

rate() first, then sum() — counter resets and aggregation across pods

Because a pod's counter drops to zero when it restarts, you compute each series' per-second rate first, which detects and repairs the drop, and only then sum across pods, since summing first hides the reset and draws a false spike.

Previously

The monitor now collects every pod's running totals; turning them into one fleet line needs the right order of operations.

Scene 05

rate() first, then sum()

Watch
Try it
Predict
Capture

What to watch for

How do you turn ever-growing counters into 'requests per second across the fleet' without the graph lying? Three checkout pods keep running totals of requests served (example values around 1,000, 1,200 and 800). A total that only climbs doesn't say how busy a pod is right now, so watch each pod's climb get turned into requests per second over a short window, then watch the three pods' numbers get added into one line for the whole checkout service.

Header: the order being used and the window length (example scrape interval 15 s).
Three checkout pods, each with two lanes: the top lane is its raw http_requests_total counter (a running total that only goes up, until the pod restarts and starts again from 0); the lower lane is its rate, the per-second increase over a window of recent samples, drawn with a bracket over the samples it uses.
A rate treats any drop as a restart and counts the new value from 0.
Bottom lane: the checkout fleet's requests per second, built by aggregation, combining many pods' series into one line, with the query written beside it. ↓ restart marks pod-2's restart at 02:00; a rose band marks a false spike.
Top-right: time-to-detect (TTD), example values.
All counts are examples.

Implementation

Highlighted lines are the ones running in the diagram right now.

Pod.restart

the counter climbs in memory and starts again from 0

1http_requests_total = 0
2def on_request():
3    http_requests_total += 1
4def on_scrape():
5    return http_requests_total
6def restart():
7    http_requests_total = 0

Query.rate

per-second increase over a window, repairing drops

1def rate(series, window):
2    samples = series.in_window(window)  # >= 4x scrape
3    if len(samples) < 2:
4        return None
5    increase = 0
6    for prev, cur in pairs(samples):
7        if cur < prev:
8            increase += cur
9        else:
10            increase += cur - prev
11    return increase / span_seconds(samples)

Query.rate_then_sum

one rate per pod series, then a single addition

1def rate_then_sum(pods, window):
2    # sum(rate(http_requests_total[1m]))
3    per_pod = []
4    for p in pods:
5        r = rate(p.series, window)
6        if r is None:
7            return None
8        per_pod.append(r)
9    return sum(per_pod)

Query.sum_then_rate

pods added per scrape first, then one rate of that series

1def sum_then_rate(pods, window):
2    # rate(sum(http_requests_total)[1m:15s])
3    combined = []
4    for t in scrape_times:
5        total = sum(p.series.at(t) for p in pods)
6        combined.append(total)
7    return rate(combined, window)

Sources

Where this sits in Metrics / Monitoring System

Scene 05 of 18, in the Ask act — rate() before sum(); add buckets, never average p99s.. A pod's counter drops to zero when it restarts, so compute each series' per-second rate first and sum after: summing the raw counters first hides the reset and draws a false spike.

Up next. Now that request counts combine honestly with rate-then-sum, the next question is whether latency percentiles combine the same way across pods.

All 18 scenes in Metrics / Monitoring System · Every curriculum

Built with Arqly

Every scene in Metrics / Monitoring System builds on the one before it.All 18 Metrics / Monitoring System scenes