Build a distributed tracing system (Jaeger / Zipkin style)

You are building the tracing system for Shopfront, the same online store whose metrics platform you built in metrics-system. A checkout that should take 200 ms is taking 3 seconds. Five teams are on the call. Every one of them has a dashboard, every dashboard is green, and nobody can prove which hop of that one request burned the time.

That is not a gap in anyone's diligence. Logs and metrics are both recorded per service, so neither can answer a question about one request across services. Fan-out amplifies the tail multiplicatively — Dean & Barroso's canonical example is 100 servers at a 1-in-100 chance of a one-second response, which puts 63% of requests over a second even though every individual server looks fine. A service's own p99 is computed over its own calls, not over the requests that touched it. And time spent queued between services appears in nobody's self-reported latency at all.

This problem builds the third pillar: one record per hop of one request, all stamped with the same id, so the slow hop names itself instead of being argued about. metrics-system gets someone paged; tracing gets them to the culprit.

One sentence runs through every stage, and it is what makes tracing harder than it looks: an incomplete trace does not look incomplete. A dropped header, an independent sampling decision, a full export queue, a late span and a skewed clock all produce a picture that is wrong and still looks finished. Nothing throws. There is no error state. The trace just quietly means something else than what it appears to mean — and the whole design exists to make that failure mode visible, bounded, or impossible.

Keep one number honest the whole way: time to culprit, the minutes between the page firing and an engineer naming the service and operation responsible.