What a million traces say together — the service dependency graph and span metrics

The parent-child edges of many traces draw the live dependency graph and counting spans by service and operation yields rates and error ratios for hops no dashboard covered — but both are only as honest as the sampling that fed them.

Previously

Reading one trace is a skill, and one trace can mislead. The same spans in bulk draw something nobody drew by hand.

Scene 15

What a million traces say together

Watch
Try it
Predict
Capture

What to watch for

What can a million traces tell you that one trace cannot? Two things, and neither of them needs a new agent. First: whenever one service's span is the parent of another service's span, that is a call across a boundary — do that match over ten thousand traces and you have drawn a map of who calls whom that nobody maintained by hand. Second: count those same spans by service, operation, kind and status and you get a request rate, an error ratio and a p99 for every hop, including hops no dashboard ever covered. You already know what a rate and a latency histogram are from the metrics curriculum; the question here is only where these ones came from. Watch the traces fold in, and keep an eye on what shows up late — and on what never shows up at all.

Left: Shopfront's services, drawn not from a wiki page but from the spans themselves — every time one service's span is the parent of another service's span, that pair becomes an arrow, and the arrow's thickness follows how many calls per second were actually observed.
That whole picture is the service dependency graph.
Right: the same spans counted by service, operation, kind and status, which turns them into a request rate, an error ratio and a p99 per operation — those counts are span metrics.
Bottom left: edges drawn against edges that really exist, and the derived checkout error rate against the true one, so the distortion is countable.
The rose chip is the queue hop: checkout hands work to orders-worker through a queue, and unless someone writes the trace context into the message there is no parent-child pair to match, so there is no edge.
Every number on screen is an example.

Implementation

Highlighted lines are the ones running in the diagram right now.

ServiceGraph.on_span

matches a client span to its server span, emits an edge

1def on_span(span, store):
2    if span.kind == CLIENT:
3        key = (span.trace_id, span.span_id)
4    elif span.kind == SERVER and span.parent_span_id:
5        key = (span.trace_id, span.parent_span_id)
6    else:
7        return                # no parent id, no key
8    peer = store.pop(key)     # keys expire at max_age
9    if peer is None:
10        return store.put(key, span)
11    client, server = order(span, peer)
12    edges[(client.service, server.service)].calls += 1

SpanMetrics.on_span

counts spans by service, operation, kind and status

1def on_span(span):
2    key = (
3        span.resource['service.name'],
4        span.name,        # an id here = one key per request
5        span.kind,
6        span.status.code,
7    )
8    if is_new(key) and len(calls) >= cardinality_limit:
9        return            # aggregation_cardinality_limit: 0
10    calls[key] += 1
11    duration[key].observe(span.end - span.start)

Collector.pipeline

both connectors run before anything is dropped

1def pipeline(span):
2    service_graph.on_span(span)
3    span_metrics.on_span(span)
4    if tail_policy.decide(span.trace_id) == DROP:
5        return
6    storage.write(span)

Sources

Where this sits in Build a distributed tracing system (Jaeger / Zipkin style)

Scene 15 of 17, in the Ship it act — What a million traces say, then design the stack.. The parent-child edges of many traces draw the live dependency graph, and counting spans by service and operation yields rates and error ratios for hops no dashboard covered — both only as honest as the sampling.

Up next. Maps and span metrics are only as honest as the sampling that fed them, and every mechanism in this curriculum now has a price. Can you pick a set for a real workload and defend each choice?

All 17 scenes in Build a distributed tracing system (Jaeger / Zipkin style) · Every curriculum

Built with Arqly

Every scene in Build a distributed tracing system (Jaeger / Zipkin style) builds on the one before it.All 17 Build a distributed tracing system (Jaeger / Zipkin style) scenes