Build a Service Mesh (Envoy / Istio style) (13 scenes)
Scene 11 · Trace and span — stitching one user request
Every sidecar emits a span tagged with the trace id from `traceparent`. One forgotten header rebuild breaks the trace silently — RED metrics keep flowing regardless.
Previously
Two sidecars on every hop, all streaming live config from one control plane, are also the only fleet-wide observers of every request — so they are exactly where you stitch one user story together.
Scene 12
Trace and span — stitching one user request
Diagram
Top: four services S1→S4 with their sidecars (the **sidecar** is the per-pod proxy that handles all request traffic; together the fleet of sidecars is the **data plane**). A request flows left-to-right; each sidecar emits a small card labeled **span** — one hop's record of start time, end time, and service. Bottom: a Gantt-style **trace** — all spans sharing one trace id, assembled into a timeline. Footnotes: the RED metrics box (rate/errors/duration, always-on counters that update independent of sampling) and access logs are the other two free observability artefacts; neither is the focus here.
↑ span — one hop's record (start, end, service)
↓ trace — all spans sharing one trace id
traceparent: 00-{trace-id}-{parent-span-id}-{flags} (W3C)
One request enters S1 and travels S1→S2→S3→S4. Each sidecar emits a span card as the dot crosses it. Below, the four spans assemble into one trace — same trace id on every span, nested by the parent it inherited from traceparent.
Implementation
Sidecar.on_inbound_request
every inbound hop becomes one span, parented by traceparent
1def on_inbound_request(req):2 tp = parse(req.headers.get('traceparent'))3 trace_id = tp.trace_id if tp else new_trace_id()4 parent_id = tp.span_id if tp else None5 span = start_span(6 name=req.path,7 trace_id=trace_id,8 parent_id=parent_id,9 )10 resp = forward_to_app(req)11 span.duration = elapsed(span.start)12 emit_to_collector(span) # span is fire-and-forget13 return resp
Sidecar.on_outbound_request
stamps traceparent on whatever request the app handed us
1def on_outbound_request(req, current_span):2 # W3C format: 00-<32 hex trace>-<16 hex span>-<2 hex flags>3 req.headers['traceparent'] = (4 f'00-{current_span.trace_id}'5 f'-{current_span.span_id}'6 f'-{flags_byte(sampled=current_span.sampled)}'7 )8 return req # sidecar can only stamp what's on the wire
Sidecar.emit (sampling vs RED)
metrics tick on every request; traces tick on flags & 0x01
1def emit_to_collector(span):2 # RED metrics: counted on EVERY request, no sampling.3 metrics.requests.inc()4 metrics.duration.observe(span.duration)5 if span.errored: metrics.errors.inc()6 # Traces: only written if the sampled bit is set.7 if span.sampled: # head decision from flags byte8 trace_store.write(span)9 # else: span is dropped at emit time.