A layer that isn't your app — collector agents and the gateway pool

A collector tier takes the jobs no application should do — stamping where the span ran, stripping anything sensitive, absorbing bursts, fanning out to more than one backend — and the whole-trace decision has to live in its central pool, because only a pool can be routed to.

Previously

The exporter ships batches to an endpoint we have not yet named. Something is on the other end of that connection, and it is doing more than writing rows.

Scene 11

A layer that isn't your app

Watch
Try it
Predict
Capture

What to watch for

What is actually on the other end of the connection the application ships its batches to? Watch one span travel there. It leaves payments knowing what it did but not where it ran; a program on the same machine stamps its pod, its node and its region; a central pool strips a card number out of it before anything leaves the cluster; and the same span is then written to two different places from one pipeline, without payments ever learning there are two. That layer — not your application, not your storage — is a collector, and none of those four jobs belonged in the application.

Top: the pipeline strip, with the collect stage opened up.
Left: Shopfront's six services, grouped by the machine each one runs on — gateway, checkout and cart on node-1; inventory, payments and ledger on node-2.
Middle: one agent per machine, then a pool of interchangeable gateways.
Right: the trace store, and an optional second backend fed from the same pipeline.
A collector is a process that is neither your application nor your storage — it receives spans, does work on them and forwards them on, deployed either as an agent beside the application or as a routable pool.
The badge on the opened collect stage names the shape you are looking at, and the chips under the last box that still sees the span show which jobs are switched on.
Below the tier: one span hop by hop, with fields it gains in green and fields struck through in rose where they were removed; the green ones are resource attributes, facts about the process a span ran in that the process cannot look up about itself.
The bar at the bottom answers the scene's one question: can any single box on this canvas see a whole trace?

Implementation

Highlighted lines are the ones running in the diagram right now.

Collector.pipeline

every batch runs the same ordered list of jobs

1def pipeline(batch):
2    if memory_used() > soft_limit:
3        return refuse(batch, retryable=True)
4    batch = enrich(batch, resource_of(self.host))
5    batch = redact(batch)
6    batch = regroup_into_batches(batch)
7    batch = tail_sample(batch)   # regroups by trace id
8    for dest in self.destinations:
9        dest.send(batch)

Agent.can_decide

a collector deployed one per machine

1# one per machine; each app exports to localhost
2def receives(self):
3    return union(p.spans for p in processes_on(self.node))
4 
5def can_decide(self, trace_id):
6    held = self.buffer[trace_id]
7    return held == every_span_of(trace_id)
8 
9def resource_of(self, host):
10    return lookup(host)  # k8s.pod, host.name, cloud.region

Gateway.can_decide

a pool with one address in front of it

1# a scaled pool of interchangeable boxes
2def route(span):
3    i = consistent_hash(span.trace_id) % len(pool)
4    return pool[i]
5 
6def receives(self):
7    return [s for s in arriving if route(s) is self]
8 
9def can_decide(self, trace_id):
10    held = self.buffer[trace_id]
11    return held == every_span_of(trace_id)

Sources

Where this sits in Build a distributed tracing system (Jaeger / Zipkin style)

Scene 11 of 17, in the Pipeline act — Off the hot path, then a tier that isn't your app.. A collector tier takes the jobs no application should do: stamping where a span ran, stripping anything sensitive, absorbing bursts, fanning out. Tail decisions live in its pool, because only a pool can be routed to.

Up next. The gateway pool is the last place a trace's spans are together in memory. Then they are written down — under what key, and what has to happen before you can read a whole trace back?

All 17 scenes in Build a distributed tracing system (Jaeger / Zipkin style) · Every curriculum

Built with Arqly

Every scene in Build a distributed tracing system (Jaeger / Zipkin style) builds on the one before it.All 17 Build a distributed tracing system (Jaeger / Zipkin style) scenes