Sampling that doesn't lie — sampling bias and adjusted counts

A uniform sample keeps proportions roughly right but destroys rare events, a per-service trace budget quietly over-represents the quiet services, and counting stored traces without multiplying each by one over its keep rate understates everything by exactly the sampling factor.

Previously

Consistent sampling gives you whole traces. It does not give you a representative answer to every question you might ask of them.

Scene 08

Sampling that doesn't lie

Watch
Try it
Predict
Capture

What to watch for

You kept 1 trace in 100. Which questions can those traces still answer honestly, and which ones can they no longer answer at all? This is one hour of Shopfront checkout traffic (example). Watch the upper bar of each pair — what really happened — and then the lower bar that a uniform 1-in-100 sampler kept, drawn on the same scale. Two of the rows shrink neatly in proportion. Watch what happens to the other two. Then watch the failure count get taken twice, once from the kept traces and once from metrics, and notice that nothing on the diagram explains why they disagree.

Four rows, one per slice of the same hour of Shopfront traffic.
Each row draws two bars on one shared scale: the upper bar is what really happened, the lower bar is what the sampler actually kept.
Because every row shares one scale, a slice that was sampled less reads as a shorter bar rather than as a smaller number.
When the sampler systematically favours or starves part of the population, so the kept traces are not a faithful miniature of reality, that is sampling bias.
Below the rows, two numbers face each other: the same failure count taken from the kept traces on the left and from metrics on the right, with the truth between them.
Under those, a chip says whether the hour's single rarest request left a trace behind.
Turning on weighting counts each kept trace as one over the rate it was kept at — its adjusted count — and draws that estimate as a dashed extension on every sample bar.

Is there a standard way to carry the keep rate along with a trace?

Partly, and it is still emerging. OpenTelemetry has a specification for putting a sampling threshold into the trace's own metadata so any downstream component can derive the adjusted count, but that document is marked Development rather than stable. It is implemented — the Collector's probabilistic sampler emits it and some SDKs read it — but different products still call the same idea different things: Honeycomb's SampleRate, Dapper's sampling coefficient, Grafana Tempo's span multiplier key. The transferable idea is the same in all of them: multiply each kept trace by one over its keep rate.

If sampling destroys rare events, why does anyone accept it?

Because at high enough throughput a genuinely notable pattern is not rare. Dapper's own words: if a notable execution pattern surfaces once in such systems, it will surface thousands of times. That defence is explicitly scoped to very high-throughput services and does not transfer to a low-traffic endpoint or to a specific named customer's failed checkout — for those, uniform sampling is simply the wrong instrument.

A fixed trace budget per service sounds obviously broken. Why is it a real strategy?

Because the alternative starves exactly the endpoints you can least afford to be blind on. If one rate has to cover a service with a 20,000-request-per-second checkout and a 2-request-per-second admin path, the busy endpoint drives the rate and the quiet one is never sampled at all — which is why Jaeger's adaptive sampler works per operation rather than per service. The cost is that raw stored-trace counts stop meaning anything, so every trace must carry its own rate.

If the count from traces can be corrected, why bother with metrics at all?

Because the corrected count is an estimate built from survivors, and its error grows precisely where the sample is thinnest. Metrics count every request at the source, so how many belongs to them. Traces answer a different question — which request, and where its time went — for the subset you happened to keep.

Implementation

Highlighted lines are the ones running in the diagram right now.

Sampler.should_sample

three ways to choose, three different p

1def should_sample(request):
2    if strategy == "uniform":
3        return keep(request, p = 0.01)
4    if strategy == "rate-limit-per-service":
5        bucket = buckets[request.service]
6        # N traces/sec, so throughput sets p
7        return keep(request, p = bucket.rate())
8    if strategy == "keep-errors-plus-baseline":
9        # needs the outcome, not known here yet
10        if request.failed:
11            return keep(request, p = 1.0)
12        return keep(request, p = 0.01)

Sampler.keep

the rate rides along with the trace it decided

1def keep(request, p):
2    if not coin_flip(p):
3        return DROP
4    trace = start_trace(request)
5    # the rate this one trace survived at
6    trace.keep_rate = p
7    store.write(trace)
8    return RECORD

Counter.estimate

counting stored traces vs counting what they stand for

1def estimate(query):
2    stored = store.select(query)
3    if not weight_by_keep_rate:
4        return len(stored)
5    total = 0
6    for trace in stored:
7        total += 1 / trace.keep_rate
8    return total
9 
10def estimate_from_one_rate(stored):
11    return len(stored) / assumed_rate

Sources

Where this sits in Build a distributed tracing system (Jaeger / Zipkin style)

Scene 08 of 17, in the Sampling act — You can't keep it all; decide once, and don't lie.. A uniform sample keeps proportions but destroys rare events; rate limiting over-represents quiet services; and counting stored traces without weighting each by the inverse of its keep rate understates everything.

Up next. Reweighting fixes the counts, but nothing can resurrect the one failure that was never kept. What if the decision waited until the trace was over?

All 17 scenes in Build a distributed tracing system (Jaeger / Zipkin style) · Every curriculum

Built with Arqly

Every scene in Build a distributed tracing system (Jaeger / Zipkin style) builds on the one before it.All 17 Build a distributed tracing system (Jaeger / Zipkin style) scenes