Design your tracing stack

A tracing stack is a chain of decisions — what to instrument, what to propagate, how much to keep and by what rule, where that decision is made, how a trace gets found, and what may be derived from it — and each is right only for a given workload.

Previously

Every mechanism in the curriculum now has a price attached. The only remaining question is which ones a given workload actually needs.

Scene 16

Design your tracing stack

Watch
Try it
Predict
Capture

What to watch for

Which of the fifteen mechanisms you have met does a given workload actually need? Here is a small one: three services, 200 requests a second, traces read by hand while debugging. A stack is already placed, and the checker on the right is green — but look at what makes each verdict trustworthy: every line names the scene it came from. Then watch a gateway tail-sampling tier get added to the design, and watch the checker's answer change without a single fact about the workload changing.

Across the top, the workload you are designing for: its service count, traffic, hops per request, and what it must be able to prove.
Down the left, your stack as six decision rows — one per choice you have been making since scene 02 — with what each one costs and buys underneath.
Down the right, the checker: one line per verdict, each carrying a chip naming the scene that justifies it.
On the third workload the checker stops scoring and refuses instead.

Implementation

Highlighted lines are the ones running in the diagram right now.

Design.check

walks the design against the workload; every verdict cites a scene

1MAX_TRACE_RETENTION = 30 * DAY   # X-Ray, fixed
2def check(design, workload):
3    if workload.retention > MAX_TRACE_RETENTION:
4        return REFUSE(
5            asked_for = 'every payment, kept forever',
6            belongs_in = 'append-only record, by account',
7        )
8    return [
9        judge_chain(design, workload),   # 03, 04
10        judge_keep(design, workload),    # 06, 07, 09, 11
11        judge_entry(design, workload),   # 10, 13, 15
12    ]

Design.judge_chain

does one request's context survive every hop it takes

1def judge_chain(design, workload):
2    for hop in workload.hops:
3        if hop.kind == 'http':
4            carried = design.header('traceparent')   # 03
5        else:
6            # no outbound request to put a header on
7            carried = design.inject_into_message()   # 04
8        if not carried:
9            return fail('the chain stops at this hop;'
10                        ' the next service starts a trace')
11    return ok()

Design.judge_keep

how much is kept, by what rule, and where that call is made

1def judge_keep(design, workload):
2    if workload.affordable_at_full_keep:           # 06
3        if design.sampling == 'tail':
4            return warn('nothing for a policy to choose')
5        return ok('keep 100%; deciding costs more')
6    out = []
7    if workload.must_keep('errors', 'over 2 s'):   # 09
8        if design.sampling != 'tail':
9            out.append(fail('decided at the first span'))
10    if design.decide_at != 'gateway_pool':         # 07, 11
11        out.append(fail('one decision, on a whole trace'))
12    return out or [ok()]

Design.sizing

the bill: span rate x hops x bytes x what you keep it for

1SPAN_BYTES = 500                 # a span on the wire
2PER_MILLION_TRACES = 5.00        # X-Ray, recorded
3def sizing(design, workload):
4    spans = workload.rps * workload.hops_per_request
5    ingest = spans * SPAN_BYTES  # tail ships all of it
6    stored_per_day = ingest * design.keep_rate * 86400
7    bill = workload.rps * 86400 / 1e6 * PER_MILLION_TRACES
8    tiers = 2 if design.decide_at == 'gateway_pool' else 1
9    held_for = min(workload.retention, MAX_TRACE_RETENTION)
10    return stored_per_day, bill, tiers, held_for

Sources

Where this sits in Build a distributed tracing system (Jaeger / Zipkin style)

Scene 16 of 17, in the Ship it act — What a million traces say, then design the stack.. A tracing stack is a chain of decisions: what to instrument, what to propagate, how much to keep and by what rule, where that decision is made, and how a trace gets found. Each is right only for a given workload.

All 17 scenes in Build a distributed tracing system (Jaeger / Zipkin style) · Every curriculum

Built with Arqly

Every scene in Build a distributed tracing system (Jaeger / Zipkin style) builds on the one before it.All 17 Build a distributed tracing system (Jaeger / Zipkin style) scenes