High cardinality is free here

A span is already one row per request per hop, so attaching the customer id adds bytes to a row that was being written anyway and multiplies nothing — the exact opposite of a metric label, where each distinct value creates another series that lives for the whole retention window.

Previously

Every hop now produces a span in the right trace, including across the queue. A span is a row that is being written anyway — so what else can ride on that row?

Scene 05

High cardinality is free here

Watch
Try it
Predict
Capture

What to watch for

A span is already being written for this hop — so what else can ride on it, and what does riding on it cost? Watch one payments span fill with rows. Each row is a key and a value; the meter at the foot of the card climbs as they land. Keep one eye on the counter to the right of the card: it stays at 1 the whole time, because this is one row, not a series. At the end, one more row lands underneath the others, with a timestamp on it.

On the left, one span from the payments service — the record of that single hop — drawn as a card filling with rows.
Each row is an attribute: a key and a value recorded on that span, like the route it served or the status it returned.
A row outlined in rose carries something that should never have been written down.
Beneath the attribute rows, under the heading EVENTS, sits a timestamped row: that is a span event — a moment during this hop, such as an exception, recorded on the span rather than as another span.
The meter at the foot of the card is the size of this one row in bytes.
On the right, the same value placed on the other side of the house: the counter for span rows against the counter for metric series.
Beneath that, a muted strip of the costs that are real here.

Implementation

Highlighted lines are the ones running in the diagram right now.

Span.set_attribute

one more key/value on a row that already exists

1def set_attribute(self, key, value):
2    # keys: lowercase, dot-delimited; your own
3    # take a com.* prefix — otel.* is reserved
4    if is_credential(key, value):
5        value = redact(value)  # url.full
6    # attr_count_limit defaults to 128
7    if len(self.attributes) >= attr_count_limit:
8        self.dropped_attributes_count += 1
9        return  # silent, but counted
10    self.attributes[key] = value
11    self.bytes += len(key) + len(value)

Span.add_event

a timestamped moment inside this hop

1def add_event(self, name, attributes, at):
2    if len(self.events) >= event_count_limit:
3        self.dropped_events_count += 1
4        return
5    self.events.append(Event(name, at, attributes))
6    # no new span_id, no new parent_span_id —
7    # the trace keeps the shape it already had
8 
9def record_exception(self, err):
10    self.add_event("exception",
11                   exception_attrs(err), clock.now())

Handler.on_charge

the customer id written to the span and to a metric

1def on_charge(span, customer_id):
2    # the same value, on both sides at once
3    span.set_attribute(
4        "com.shopfront.customer_id", customer_id)
5    charges.add(1, {"customer_id": customer_id})
6 
7# what charges.add does in the metrics SDK
8def add(self, n, labels):
9    key = fingerprint(labels)
10    if key not in self.series:
11        self.series[key] = new_series(key)
12    self.series[key].value += n

Sources

Where this sits in Build a distributed tracing system (Jaeger / Zipkin style)

Scene 05 of 17, in the Carry it act — The header, the queue that drops it, what rides along.. A span is already one row per request per hop, so attaching a customer id adds bytes to a row that was being written anyway and multiplies nothing — the exact opposite of a metric label.

Up next. Attributes are nearly free per span, but a span itself is not free, and there is one per hop per request. What does keeping every one of them actually cost?

All 17 scenes in Build a distributed tracing system (Jaeger / Zipkin style) · Every curriculum

Built with Arqly

Every scene in Build a distributed tracing system (Jaeger / Zipkin style) builds on the one before it.All 17 Build a distributed tracing system (Jaeger / Zipkin style) scenes