Decide after you've seen the whole trace — tail-based sampling and the decision window

To judge a trace on whole-trace facts you must route every span of that trace to the same decision point, hold them for a window, and accept that anything arriving after the window is judged on its own.

Previously

A head decision is made before anyone knows how the request turned out. The only way to keep exactly the interesting traces is to decide after the fact.

Scene 09

Decide after you've seen the whole trace

Watch
Try it
Predict
Capture

What to watch for

How do you keep exactly the traces worth keeping, when whether a trace is worth keeping — did anything error, was the whole checkout over 2 s — is only knowable after the request is over? You wait. Watch six spans of each trace arrive from Shopfront's services and get handed to two decision points, which hold them instead of writing them straight to storage. Count the filled segments on each box as the spans land, and watch both memory meters at the same time.

Top: the pipeline strip, with the sample stage opened up.
Left: spans arriving from Shopfront's services, six per trace, and the routing box that picks which decision point each span goes to.
Middle: two decision points, each printing the traces it is holding as segmented pills — filled segments are the spans that have actually reached it out of the six the trace has, so orders-worker's span fills in only once the wait outlasts the +45 s marker on the axis — plus a meter for how much of its 50,000-trace buffer is in use.
Below: the wait on a time axis, with orders-worker's spans marked at +45 s.
Deciding what to keep only after the trace is over is tail-based sampling; the fixed wait before judging is the decision window.

Implementation

Highlighted lines are the ones running in the diagram right now.

Router.route

the front tier picks which box a span goes to

1# routing_key: traceID is the default for traces
2def route(span, boxes):
3    if routing_key == "round_robin":
4        i = next_index()       # whoever is next
5    else:
6        i = consistent_hash(span.trace_id) % len(boxes)
7    boxes[i].send(span)
8 
9# membership change moves ~R/N of the trace ids

TailSampler.on_span

hold every arriving trace, keep or not

1decision_wait = 30s      # default
2num_traces = 50_000      # circular buffer
3 
4def on_span(span):
5    tid = span.trace_id
6    if tid not in buffer:        # judged ones were popped
7        if len(buffer) == num_traces:
8            evict_oldest()   # ..._dropped_too_early
9        buffer[tid] = new_trace()
10        start_timer(tid, decision_wait)
11    buffer[tid].spans.append(span)

TailSampler.decide

the timer fires; judge what is in the buffer

1def decide(tid):
2    trace = buffer.pop(tid)
3    spans = trace.spans
4    kept = False
5    if "status_code" in policies:
6        kept |= any(sp.status == ERROR for sp in spans)
7    if "latency" in policies:
8        kept |= duration(spans) > 2s
9    if "probabilistic" in policies:
10        kept |= hash(tid) < 0.01
11    if kept:
12        storage.write(spans)

Sources

Where this sits in Build a distributed tracing system (Jaeger / Zipkin style)

Scene 09 of 17, in the Sampling act — You can't keep it all; decide once, and don't lie.. To judge a trace on whole-trace facts you must route every span of it to the same decision point, hold them for a window, and accept that anything arriving later is judged on its own — the queue hop again.

Up next. Deciding late means every span of one trace has to meet somewhere outside the application. Before a span can reach anywhere at all, it has to leave the process — without making the customer wait.

All 17 scenes in Build a distributed tracing system (Jaeger / Zipkin style) · Every curriculum

Built with Arqly

Every scene in Build a distributed tracing system (Jaeger / Zipkin style) builds on the one before it.All 17 Build a distributed tracing system (Jaeger / Zipkin style) scenes