Build a distributed logging stack (ELK / Loki)

Ship lines off N hosts, choose what to index, age data through tiers, retain or delete on schedule, and survive a chatty service — built one decision at a time.

SavedSaved on this device

The problem

Ship lines off N hosts, choose what to index, age data through tiers, retain or delete on schedule, and survive a chatty service — built one decision at a time.

You are building the simplest stack that can take log lines emitted by hundreds of pods on dozens of hosts, get them off-host before they age out of the on-host file, store them somewhere a query can reach, and answer "show me every ERROR matching 'connection refused' in the last hour" in a time the on-call engineer is willing to wait. Clients emit lines (log.info(...)); the system tails, batches, ships, parses, indexes, tiers, retains, and queries them; somewhere in the middle a thoughtful operator decides which lines are worth indexing and which can stay opaque.

ELK and Loki are the canonical worked examples — opposite ends of the same trade-off curve. Once you build the core, the design space of every logging stack collapses to a handful of named trades — pay-at-write vs pay-at-read, the cardinality ceiling you defend, the cost ladder you age through, and where you put the line between "we keep it" and "we sampled it away."

Resist the urge to "describe ELK" or "describe Loki." Make decisions yourself, defend them, and let the workload push back.

What you will cover

the shipping agent: tail offset, batch, retry, backpressure
block / drop / spill: how a backend outage starves application threads
structured logs: fields (ELK) vs labels (Loki), parse-at-write vs parse-at-read
the indexing fork: Lucene inverted index (heavy index, fast read) vs Loki labels-only index + chunks (cheap write, grep-on-read)
execution-plan asymmetry: posting-list intersection vs label-resolve-then-stream-from-S3
cardinality: low-card → labels/fields, high-card → body / structured metadata
tiering: hot/warm/cold/frozen ILM and the two-orders-of-magnitude cost ladder
retention vs deletion: tombstones, force-merge lag, the PII per-template leak
head sampling vs tail sampling: when uniform sampling drops the only error
backend distribution: distributors, ingesters, queriers — partition by hash(doc_id) vs hash(label-set)

Reading

Grafana Labs — 'Loki: Like Prometheus, but for logs' (announcement, 2018)
Elastic — Data tiers and Index Lifecycle Management (elastic.co/guide)
Grafana — Loki labels and cardinality best practices (grafana.com/docs/loki)
Fluent Bit — Backpressure and storage configuration (docs.fluentbit.io)
Filebeat — Internal queue tuning (elastic.co/guide/en/beats)
Promtail — Positions, WAL, and the checkpoint-before-send gotcha (grafana.com/docs/loki)
OpenTelemetry — Sampling strategies (head vs tail) (opentelemetry.io)
Designing Data-Intensive Applications, Kleppmann — Ch 3 (storage), Ch 11 (stream processing)

About Build a distributed logging stack (ELK / Loki)

Ship lines off N hosts, choose what to index, age data through tiers, retain or delete on schedule, and survive a chatty service — built one decision at a time.

Difficulty: intermediate
Time: about 80 minutes
Stages: 9
Topic: Observability: Metrics, Logs & Traces

How this problem is worked

Nine stages, from what the thing is for to how it compares with the real implementations. Each asks one question, and the simulator runs the architecture you draw against the requirements you wrote.

01Purpose & invariantsWhat is this for, and what must always be true of it?
02Workload characterizationWho writes, who reads, and in what shapes?
03Data model & on-disk formatWhat does the data look like at rest?
04Core algorithmsHow do the write path and the read path actually work?
05Distribution & replicationHow does this scale out and survive losing a machine?
06Consistency & correctnessUnder concurrency and failure, what is guaranteed?
07Failure modes & recoveryWhat actually happens when each part fails?
08Operational characteristicsCan a human run this at three in the morning?
09Trade-offs & comparisonWhere does this sit against the alternatives?

Primary sources for this problem

Grafana Labs — 'Loki: Like Prometheus, but for logs' (announcement, 2018)
Elastic — Data tiers and Index Lifecycle Management (elastic.co/guide)
Grafana — Loki labels and cardinality best practices (grafana.com/docs/loki)
Fluent Bit — Backpressure and storage configuration (docs.fluentbit.io)
Filebeat — Internal queue tuning (elastic.co/guide/en/beats)
Promtail — Positions, WAL, and the checkpoint-before-send gotcha (grafana.com/docs/loki)
OpenTelemetry — Sampling strategies (head vs tail) (opentelemetry.io)
Designing Data-Intensive Applications, Kleppmann — Ch 3 (storage), Ch 11 (stream processing)

More in Observability: Metrics, Logs & Traces

The three pillars, built rather than bought: a time-series database, a log pipeline, a tracing system, and the metrics platform on top.

Browse the full problem catalog, or see what the simulator does and does not model.