Every request, every hop, forever? — full capture cost and the keep rate

Spans multiply — requests per second times hops per request — so full capture is a storage bill measured in rows per day plus a measured latency cost at the source, which makes traces the one pillar where you deliberately keep a fraction.

Previously

An attribute costs bytes on a row that already exists. The question that raises is how many rows there are.

Scene 06

Every request, every hop, forever?

Watch
Try it
Predict
Capture

What to watch for

How much does it actually cost to record every request, at every hop, and keep it? Shopfront takes about 2,000 checkout-path requests a second (example), and each one crosses 8 services — so each one is 8 spans, not 1. Watch the chain multiply: requests per second, then hops, then bytes per stored span. The last card is what lands on disk every day, and the dashed line is the team's daily allowance. Keep an eye on the amber strip at the bottom too: the second cost of recording everything is not paid on disk at all.

A left-to-right meter chain.
Each card is one stage of the same arithmetic: Shopfront's requests per second, multiplied by the number of hops each request makes, gives spans per second; multiplied again by the bytes one stored span occupies, that gives bytes per day.
The pill on each arrow is the factor being applied.
Below the chain, the left meter is the keep rate — the fraction of traces you let through — and the right meter is what that leaves on disk each day, against a dashed budget line.
Under them, one mark per failed checkout from yesterday: solid where a trace was kept, dashed where none was.
Deliberately keeping only a fraction of traces is called sampling.
The amber strip at the bottom is the cost paid on the request path itself.

Implementation

Highlighted lines are the ones running in the diagram right now.

Capacity.raw_per_day

what recording every hop costs before any choice

1BYTES_PER_SPAN = 500   # Tempo plans ~500 B; Dapper: 426 B
2SECONDS_PER_DAY = 86_400
3 
4def raw_per_day(requests_per_sec, hops_per_request):
5    spans_per_sec = requests_per_sec
6    spans_per_sec *= hops_per_request   # one span per hop
7    bytes_per_sec = spans_per_sec * BYTES_PER_SPAN
8    return bytes_per_sec * SECONDS_PER_DAY
9 
10raw = raw_per_day(rate_into_checkout, services_on_path)

Capacity.stored_per_day

the one fraction that divides the bill, and what else it divides

1DAILY_BUDGET = 100 * GB
2 
3def stored_per_day(raw, keep):
4    # sampling: record only this fraction of traces
5    return raw * keep
6 
7def over_budget(raw, keep):
8    return stored_per_day(raw, keep) > DAILY_BUDGET
9 
10def still_have_a_trace(rare_events, keep):
11    return rare_events * keep

BatchProcessor.on_end

the second line of the bill, paid in latency

1def on_end(span):              # still on the request path
2    buf = serialize(span)      # the term that costs
3    queue.put(buf)             # a background thread batches
4 
5# the span object itself: 204 ns root, 176 ns non-root
6# Dapper's web-search cluster, measured per rate:
7#   1/1    +16.3% latency,  -1.48% throughput
8#   1/8    +4.12%,          -0.23%
9#   1/16   +2.12%,          -0.08%

Sources

Where this sits in Build a distributed tracing system (Jaeger / Zipkin style)

Scene 06 of 17, in the Sampling act — You can't keep it all; decide once, and don't lie.. Spans multiply: requests per second times hops per request. Full capture is a storage bill plus a measured latency cost on the request path, which makes traces the one pillar where you deliberately keep a fraction.

Up next. You cannot keep everything, so you have to keep a fraction. Who decides which fraction — and what happens if every service decides for itself?

All 17 scenes in Build a distributed tracing system (Jaeger / Zipkin style) · Every curriculum

Built with Arqly

Every scene in Build a distributed tracing system (Jaeger / Zipkin style) builds on the one before it.All 17 Build a distributed tracing system (Jaeger / Zipkin style) scenes