Scene 06

You can't average a p99

Per-pod percentiles can't be combined, but histogram buckets are plain counters that add up across pods, so the percentile is estimated after summing, with bucket boundaries setting accuracy and each bucket costing a series.

Previously

Counts combine by rate then sum; latency is the harder case, because a percentile isn't a count.

Scene 06

You can't average a p99

Watch
Try it
Predict
Capture

What to watch for

How slow is checkout for the unluckiest users, across many pods? An average latency won't say — the fast requests drag it to the middle. Watch three checkout pods appear: pod-a and pod-b answer in about 100 ms, pod-c in about 2 s. Each pod reports the latency that 99 of every 100 of its own requests beat — that number is its p99, a percentile — and beside the three badges sits the fleet's own p99, computed over all 1,000 requests. Read those two numbers against each other before you continue.

Three checkout pods, each with the number of requests it served, a thin traffic-share bar, and a small bar chart counting its requests per latency range.
Each pod's badge shows its p99 — a percentile: the latency that 99 out of every 100 of its requests beat.
The middle band shows how the fleet number is being combined right now.
The wide chart adds the pods' bucket counters together; that set of counters is the histogram: one counter per latency bucket (le = 'less than or equal'), plus _sum and _count.
A violet estimate marker sits on that summed chart, showing where inside its bucket the estimated p99 falls.
The panel beside it gives the fleet p99 three ways — the average of the pods' badges (amber), the estimate read off the summed buckets (violet), and the true p99 over all 1,000 requests (green).
The chip underneath counts the series each pod pays for.
All counts and bucket layouts are examples.

Implementation

Highlighted lines are the ones running in the diagram right now.

App.observe

every request bumps each bucket whose bound it beats

1http_duration = Histogram(
2    "http_request_duration_seconds",
3    buckets=LE_BOUNDS,
4)
5def observe(seconds):
6    for le in LE_BOUNDS:
7        if seconds <= le:
8            _bucket[le] += 1       # le="0.5"
9    _bucket["+Inf"] += 1
10    _sum += seconds
11    _count += 1

Query.fleet_quantile

histogram_quantile(0.99, sum by (le) (rate(_bucket[5m])))

1def fleet_quantile(q):
2    total = {}
3    for le in LE_BOUNDS + ["+Inf"]:
4        total[le] = sum(rate(p._bucket[le]) for p in pods)
5    rank = q * total["+Inf"]
6    lo, seen = 0, 0
7    for le, count in sorted(total.items()):
8        if count >= rank:
9            frac = (rank - seen) / (count - seen)
10            return lo + (le - lo) * frac
11        lo, seen = le, count

Query.average_of_p99s

each pod's own p99, divided by how many pods there are

1def pod_p99(p):
2    xs = sorted(p.recent_latencies)
3    return xs[int(0.99 * len(xs))]
4 
5def average_of_p99s():
6    badges = [pod_p99(p) for p in pods]
7    return sum(badges) / len(badges)  # len(badges) = 3

Sources

Where this sits in Metrics / Monitoring System

Scene 06 of 18, in the Ask act — rate() before sum(); add buckets, never average p99s.. Per-pod percentiles can't be combined, but histogram buckets are plain counters that add up across pods, so the fleet percentile is estimated after summing — and bucket boundaries set its accuracy.

Up next. Now that we can compute honest error rates and latency across the fleet, the next question is how one of those numbers becomes a page without paging on every blip.

All 18 scenes in Metrics / Monitoring System · Every curriculum

Built with Arqly

Every scene in Metrics / Monitoring System builds on the one before it.All 18 Metrics / Monitoring System scenes