Decide once, carry the answer — head-based sampling and the sampled flag

Sampling is per trace, not per span: if N hops each flip their own coin at probability p, a complete trace survives with probability p to the power N, and what you store is mostly fragments that look like whole traces.

Previously

Keeping a fraction is settled. Who makes that call, and where, is not — and the obvious answer is that each service decides what it can afford.

Scene 07

Decide once, carry the answer

Watch
Try it
Predict
Capture

What to watch for

If you are only going to keep some traces, who makes that call — and where? The obvious answer is the one every team reaches for first: let each service keep the fraction it can afford. So all five hops of one Shopfront checkout are set to 10%, and each one flips its own coin the moment the request lands on it. Watch the lanes thin, one hop at a time, and watch the two counters at the foot: how many checkouts end up recorded at every single hop, and how many end up recorded at some hops but not others. You already met keeping-a-fraction when we sampled log lines in the logging curriculum, and none of that is re-derived here. Two things are new: the unit is a whole trace rather than a single line, and every hop has to land on the same answer as every other hop.

One lane per hop of a single Shopfront checkout, left to right.
Each lane is a grid of about 100 cells standing for that hop's traffic — solid cells were recorded, dashed cells were not.
The wire across the top is the request travelling between hops: it either carries a decision or reads 'decides again', meaning that hop is about to make its own call.
Every one of these calls is made the instant the request arrives, before anyone knows whether it was slow or broke — deciding at the head of the request like that is head-based sampling.
The dashed funnel under the lanes is all the traffic; the ribbon inside it is the share of requests that were recorded at every hop so far.
The strip beneath prints the arithmetic, and the two cards at the foot are the whole scene: complete traces on the left, stumps on the right.

Implementation

Highlighted lines are the ones running in the diagram right now.

Sampler.should_sample

each hop flips its own coin, ignoring the caller

1def should_sample(parent_ctx, trace_id):
2    # parent_ctx is never read, so a hop cannot
3    # see what any other hop decided
4    return random() < self.ratio
5 
6base     = self.ratio       # configured per service
7exponent = len(path)        # one call per hop
8whole    = base ** exponent

ParentBased.should_sample

the root decides, every later hop obeys the flag

1def should_sample(parent_ctx, trace_id):
2    if parent_ctx is None:      # only hop 1 lands here
3        return self.root.should_sample(trace_id)
4    if parent_ctx.sampled:      # the bit off traceparent
5        return True             # record, unconditionally
6    return False
7 
8# self.ratio is an argument to self.root, so a hop
9# that already has a parent never consults it
10# one decision per trace, whatever len(path) is
11# OTEL_TRACES_SAMPLER=parentbased_traceidratio

TraceIdRatio.should_sample

same id, same ratio, same answer at every hop

1def should_sample(parent_ctx, trace_id):
2    # every hop already carries the trace id
3    r = randomness_from(trace_id)   # 0.0 to 1.0
4    return r < self.ratio
5 
6# no coin, no flag, nothing negotiated: the same id
7# and the same ratio give the same answer at every
8# hop on the path
9# so every service has to be given the same ratio
10# OTEL_TRACES_SAMPLER=traceidratio

Sources

Where this sits in Build a distributed tracing system (Jaeger / Zipkin style)

Scene 07 of 17, in the Sampling act — You can't keep it all; decide once, and don't lie.. Sampling is per trace, not per span. If N hops each flip their own p coin a whole trace survives with probability p to the N, and what you store is mostly fragments that look like whole traces.

Up next. One decision per trace gives you whole traces. Now that the traces you kept are real, what can you honestly say about the ones you did not keep?

All 17 scenes in Build a distributed tracing system (Jaeger / Zipkin style) · Every curriculum

Built with Arqly

Every scene in Build a distributed tracing system (Jaeger / Zipkin style) builds on the one before it.All 17 Build a distributed tracing system (Jaeger / Zipkin style) scenes