Nobody knows the trace id

Every way into the store other than the id costs something — a second structure over span fields prices high-cardinality attributes the way a log index does, a scan trades that storage for query compute — so the cheapest way in is to arrive already holding the id.

Previously

Get-by-id is one cheap lookup. The engineer looking at a page does not have an id — they have a symptom and a time range.

Scene 13

Nobody knows the trace id

  1. Watch
  2. Try it
  3. Predict
  4. Capture
three routes to the same tracethe index answers fast, and the index is the thing you pay forindexed: service · operation · statustime range: last 1 hour1 · search a span indexchosenquestion: slow 500sspan index lookuptrace ids come backget-by-id eachevery indexed attribute is another structure to write and keepanswer time · s (example)1.2 / 12span index: curated columns42 / 100budget2 · scan the blocks yourselfwindow: last 1 hourquestion: slow 500sprune by timecheck block filterread what is leftcheap to store, and the query pays for it every single timescan time · s (example)5.6 / 12no index stored — pay per query3 / 1003 · arrive holding the idp99 bucket spikesbucket carries a trace idget-by-id · no searchthe trace id was recorded on the metric when the request ranclick to trace · s (example)0.4 / 12nothing extra stored0 / 100the tradelatency falls exactlyas stored index rises1 · search a span index2 · scan the blocks yourself3 · arrive holding the idlatencystored indexScene 05: customer_id was free to put on a span. It is not free to index — every indexed field is another structure to write, compact, back up and get paged about.Route 1 asks a structured question — checkout, over 2 s, status 500, last hour — and a span index answers it in about a second.
What to watch for

It is 3 a.m., the checkout alert just fired, and you have a symptom and a time range — not a trace id. So how do you find the one trace that matters? Watch three routes go after the same 3-second Shopfront checkout. The first searches a second structure the store keeps over span fields — service, operation, status, duration — which is called a span index. The second keeps no such structure and reads the raw blocks in the window. The third does not search at all.

Continue unlocks when the animation finishes.
Implementation

Highlighted lines are the ones running in the diagram right now.

Index.write
what nominating an attribute costs on every span written
# a budget decision, not a default:
INDEXED_FIELDS = intrinsics + nominated_attrs
# intrinsics = service, operation, status, duration
def write_span(span):
blocks.append(span)
for field in INDEXED_FIELDS:
value = span.get(field)
if value is None: continue
# one posting per distinct value of field
index.add(field, value, span.trace_id)
# index compacts and is backed up with the blocks
Search.by_attribute
lane 1 — ask the structure, never touch the spans
def by_attribute(question, window):
if not question.fields <= INDEXED_FIELDS:
return CANNOT_ANSWER
postings = index.lookup(
question.fields, question.values,
)
# window narrows postings; it reads nothing more
ids = [p.trace_id for p in postings
if p.time in window]
return [get_by_id(tid) for tid in ids]
Search.scan_blocks
lane 2 — keep no structure, read the window instead
def scan_blocks(question, window):
candidates = []
for block in blocklist:
# every block records its min/max time
if block.max_time < window.start: continue
if block.min_time > window.end: continue
if not block.stats.may_contain(question):
continue
candidates.append(block)
# thousands of short-lived jobs, a slice each
return fan_out(candidates, read_and_match)
Exemplar.open
lane 3 — the id was attached long before the page fired
# wired up on the metrics side, months ago
def observe(duration, ctx):
bucket = histogram.bucket_for(duration)
bucket.count += 1 # the value counts here too
bucket.exemplars.append(Exemplar(
trace_id=ctx.trace_id,
span_id=ctx.span_id,
value=duration,
)) # up to 5 per data point
def open_from_p99(bucket): # no search runs
return get_by_id(bucket.exemplars[0].trace_id)

Where this sits in Build a distributed tracing system (Jaeger / Zipkin style)

Scene 13 of 17, in the Store & read act — Key by trace id, find it, read it, distrust it.. Every way into the store other than the id costs something: an index prices high-cardinality attributes, a scan trades that storage for query compute. The cheapest way in is to arrive already holding the id.

Up next. You are now holding the right trace. What do you actually look at to name the culprit in under five minutes?

All 17 scenes in Build a distributed tracing system (Jaeger / Zipkin style) · Every curriculum

Built with Arqly
Every scene in Build a distributed tracing system (Jaeger / Zipkin style) builds on the one before it.All 17 Build a distributed tracing system (Jaeger / Zipkin style) scenes