Where the chain breaks: queues, pools, batches — context injection and span links across queues

A queue hop has no request to attach a header to, so the context must be written into the message itself — and because one consumer batch can have many causes while a span has exactly one parent, the consumer records a link to each cause instead.

Previously

Copying a header works while one service calls another over the wire. checkout does not call orders-worker — it publishes a message and returns.

Scene 04

Where the chain breaks: queues, pools, batches

Watch
Try it
Predict
Capture

What to watch for

What happens to the chain when a service never calls the next one at all? checkout does not make a request to orders-worker — it drops a message on a queue and returns to the customer. (How a queue stores and hands out messages is the build-kafka curriculum's job; here it is simply a barrel that holds messages.) There is no outbound request, so there is nothing to put a header on. Watch what the two halves of this checkout end up looking like.

Top row: the Shopfront path — gateway → checkout, then checkout → orders-worker through the rose queue barrel.
Left card: the queue, with one cell per waiting message; a solid tinted cell carries the trace context, a dashed italic cell carries nothing.
Writing the context into the message before publishing is context injection.
Right column: the trace(s) that result, on one shared clock — a rose-framed trace is a second, rootless trace nobody asked for.
Bottom band, titled 'what the consumer span hangs off' — the consumer span is just the one span the worker records while it processes the batch it took off the queue.
One parent box, and an arrow per recorded cause; each recorded cause is a span link.
The clock top-right is time-to-culprit (example numbers).

Implementation

Highlighted lines are the ones running in the diagram right now.

Producer.publish

checkout writes the context into the order it sends

1def publish(destination, order):
2    span = tracer.start_span(
3        f"publish {destination}", kind = PRODUCER,
4    )
5    msg = Message(body = order)
6    with use_context(span):
7        propagator.inject(msg.headers)  # into the message
8    broker.send(destination, msg)
9    span.end()
10    return  # checkout has answered the customer

Consumer.poll_loop

the worker takes a batch and reads each message's context

1def poll_loop():
2    loop forever:
3        msgs = broker.poll(max_messages)
4        origins = []
5        for m in msgs:
6            # extracted at process time, not at publish time
7            ctx = propagator.extract(m.headers)
8            if ctx.is_valid():
9                origins.append(ctx)
10        on_batch(msgs, origins)

Consumer.on_batch

building the one span that covers the whole batch

1def on_batch(msgs, origins):
2    parent = current_context()
3    if len(msgs) == 1 and origins:
4        # MAY — single message only; a link also works
5        parent = origins[0]
6    span = tracer.start_span(
7        f"process {destination}", kind = CONSUMER,
8        parent = parent,  # exactly one
9        # span_limits.max_links defaults to 128
10        links = [Link(c) for c in origins],
11    )
12    with use_context(span): handle_all(msgs)

Sources

Where this sits in Build a distributed tracing system (Jaeger / Zipkin style)

Scene 04 of 17, in the Carry it act — The header, the queue that drops it, what rides along.. A queue hop has no request to attach a header to, so the context must ride inside the message. And because one batch can have many causes while a span has exactly one parent, the consumer records a link to each.

Up next. The context now reaches every hop, including the one across the queue, so every hop produces a span in the right trace. What else is worth writing onto that span?

All 17 scenes in Build a distributed tracing system (Jaeger / Zipkin style) · Every curriculum

Built with Arqly

Every scene in Build a distributed tracing system (Jaeger / Zipkin style) builds on the one before it.All 17 Build a distributed tracing system (Jaeger / Zipkin style) scenes