Never make a customer wait for telemetry

A finished span goes into a bounded in-memory queue that a background worker drains in batches, and when that queue is full the library drops spans and counts the drops — because blocking the request instead would turn a backend slowdown into an application outage.

Previously

We asked every span of a trace to travel to a shared decision point. The first leg of that journey is out of the application itself, and the application is serving a customer.

Scene 10

Never make a customer wait for telemetry

Watch
Try it
Predict
Capture

What to watch for

A span is finished — so how does it get out of the application and over to wherever traces are kept, without the customer who is still waiting for a reply paying for that trip? Watch the two lanes. On the left, the handler records the span and replies immediately. On the right, the finished spans pile into a fixed-size holding area that a background worker drains in batches. Then traffic bursts, the worker falls behind, and the holding area runs out of room. Watch what the human sees at the bottom afterwards.

Top: the pipeline strip, with the export stage opened up.
Left, in blue: the request path Shopfront's customer is waiting on — the handler does its work, records the span, and replies — with the latency bar underneath split into the work itself and whatever recording the span added.
Right: the fixed-size holding area the finished spans go into, drawn as slots (each drawn slot stands for 256 spans), and the background worker that pulls batches off it and ships them to the telemetry backend.
That holding area is the span queue, and the worker that drains it is the exporter.
This queue lives inside the application's own memory and is not Shopfront's rose message queue between checkout and orders-worker.
Below the lanes: the counter of spans that had nowhere to go, and the trace a human opens afterwards — dashed slate bars are spans that never arrived, and nothing anywhere is marked as an error.

Implementation

Highlighted lines are the ones running in the diagram right now.

SimpleSpanProcessor.on_end

one network call per span, inside the request

1def handle_request(req):
2    spans = do_work(req)   # 4 spans per request
3    for span in spans:
4        processor.on_end(span)
5    return response        # +4 x export_rtt
6 
7def on_end(span):          # SimpleSpanProcessor
8    resp = exporter.export([span])  # one OTLP call
9    block_until(resp)

BatchSpanProcessor.on_end

the handler appends once and returns

1MAX_QUEUE_SIZE = 2048   # spans, in app memory
2 
3def on_end(span):       # BatchSpanProcessor
4    if len(queue) >= MAX_QUEUE_SIZE:
5        spans_dropped.add(
6            1, {"error.type": "queue_full"},
7        )
8        return
9    queue.append(span)

BatchSpanProcessor.worker

a background thread drains the queue in batches

1def worker():                  # background thread
2    while running:
3        wait(scheduled_delay_ms=5000)
4        while queue:
5            batch = queue.take(max_export_batch=512)
6            resp = exporter.export(batch)  # OTLP
7            wait(resp, timeout_ms=30000)

Sources

Where this sits in Build a distributed tracing system (Jaeger / Zipkin style)

Scene 10 of 17, in the Pipeline act — Off the hot path, then a tier that isn't your app.. A finished span goes into a bounded queue that a background worker drains in batches. When it fills, the library drops spans and counts them — blocking the request would turn a backend slowdown into an outage.

Up next. The application now hands batches to something else and forgets them. What is that something else, and what else should it be doing?

All 17 scenes in Build a distributed tracing system (Jaeger / Zipkin style) · Every curriculum

Built with Arqly

Every scene in Build a distributed tracing system (Jaeger / Zipkin style) builds on the one before it.All 17 Build a distributed tracing system (Jaeger / Zipkin style) scenes