The write path: one row per span, one key per trace

Spans are written as they arrive under a random trace id that spreads perfectly across partitions, nothing ever closes a trace, and the tree is rebuilt in the viewer — so a partial trace is the normal steady state, not an error.

Previously

The gateway hands spans to a store one at a time, as they arrive, with no notion of which trace is finished.

Scene 12

One row per span, one key per trace

Watch
Try it
Predict
Capture

What to watch for

Where do spans actually go, and who puts a trace back together? Watch two Shopfront requests' spans arrive at the store over 45 seconds — interleaved, out of order, each row written the instant its own process finished that span. Keep your eye on the right-hand half of the canvas while they land.

A dashed line splits WRITE TIME from READ TIME.
Left is write time: the arrival rail, one lane per trace, showing each span row landing at the store the moment its own process finished it — two requests interleaved, in no particular order.
The chip under the rail is the key the store files each row under, and the bins below are the storage partitions that key spreads rows across; filing by the random trace id is a trace-id partition — every row of one trace in one bin, and the bins level because the ids are random.
The meter and ruler are how much is stored and for how long.
Right is read time: empty until somebody asks, then the rows under one key are pulled back and joined into a tree by their parent pointers — that join, done only when a read happens and only out of the rows that exist at that instant, is read-time assembly, and the right-hand card is the only place in the system where it happens.
Nothing on the write side ever draws a tree.

Implementation

Highlighted lines are the ones running in the diagram right now.

TraceStore.write

one row, filed under the key, the moment it arrives

1def write(span):
2    if partition_key == "trace_id":
3        key = span.trace_id      # 128-bit random
4    else:
5        key = span.service_name  # a few values
6    partition_of(key).append(row(
7        trace_id, span_id, parent_span_id,
8        start_us, duration_us, attributes,
9    ), ttl = retention)
10    # nothing follows: no assemble(),
11    # no mark_complete(), no wait_for_siblings()

Viewer.read_trace

group the rows sharing one id, join child to parent

1def read_trace(trace_id):
2    rows = store.get(key = trace_id)  # 1 partition
3    node = {r.span_id: Node(r) for r in rows}
4    roots = []
5    for r in rows:
6        parent = node.get(r.parent_span_id)
7        if parent: parent.children.append(node[r.span_id])
8        else:      roots.append(node[r.span_id])
9    return roots
10# keyed by service_name, this get() becomes a
11# fan-out scan of every partition

TraceStore.expire

the only thing in the system that removes a row

1retention = {                  # all three real
2    "jaeger on cassandra":  48h,  # trace TTL
3    "tempo blocks":        336h,  # block_retention
4    "aws x-ray":           720h,  # fixed, no knob
5}
6def expire():
7    for row in partition.scan():
8        if age(row) > retention:
9            row.drop()   # one row, not one trace
10    # stored_bytes = write_rate * retention

Sources

Where this sits in Build a distributed tracing system (Jaeger / Zipkin style)

Scene 12 of 17, in the Store & read act — Key by trace id, find it, read it, distrust it.. Spans are written as they arrive under a random trace id that spreads perfectly across partitions. Nothing ever closes a trace, so a partial trace is the normal steady state and the tree is rebuilt at read time.

Up next. Reading a trace is one cheap lookup — if you have the id. Nobody has the id. So how do you find the trace that matters?

All 17 scenes in Build a distributed tracing system (Jaeger / Zipkin style) · Every curriculum

Built with Arqly

Every scene in Build a distributed tracing system (Jaeger / Zipkin style) builds on the one before it.All 17 Build a distributed tracing system (Jaeger / Zipkin style) scenes