#16Build a distributed logging stack (ELK / Loki)
Ship lines off N hosts, choose what to index, age data through tiers, retain or delete on schedule, and survive a chatty service — built one decision at a time.

You are building the simplest stack that can take log lines emitted by hundreds of pods on dozens of hosts, get them off-host before they age out of the on-host file, store them somewhere a query can reach, and answer "show me every ERROR matching 'connection refused' in the last hour" in a time the on-call engineer is willing to wait. Clients emit lines (log.info(...)); the system tails, batches, ships, parses, indexes, tiers, retains, and queries them; somewhere in the middle a thoughtful operator decides which lines are worth indexing and which can stay opaque.

ELK and Loki are the canonical worked examples — opposite ends of the same trade-off curve. Once you build the core, the design space of every logging stack collapses to a handful of named trades — pay-at-write vs pay-at-read, the cardinality ceiling you defend, the cost ladder you age through, and where you put the line between "we keep it" and "we sampled it away."

Resist the urge to "describe ELK" or "describe Loki." Make decisions yourself, defend them, and let the workload push back.

Builds on: Consistent hashing & the ring — 2-min primers appear where needed.
Reading: Grafana Labs — 'Loki: Like Prometheus, but for logs' (announcement, 2018) · Elastic — Data tiers and Index Lifecycle Management (elastic.co/guide) · Grafana — Loki labels and cardinality best practices (grafana.com/docs/loki) · Fluent Bit — Backpressure and storage configuration (docs.fluentbit.io) · Filebeat — Internal queue tuning (elastic.co/guide/en/beats) · Promtail — Positions, WAL, and the checkpoint-before-send gotcha (grafana.com/docs/loki) · OpenTelemetry — Sampling strategies (head vs tail) (opentelemetry.io) · Designing Data-Intensive Applications, Kleppmann — Ch 3 (storage), Ch 11 (stream processing)
the shipping agent: tail offset, batch, retry, backpressure
block / drop / spill: how a backend outage starves application threads
structured logs: fields (ELK) vs labels (Loki), parse-at-write vs parse-at-read
the indexing fork: Lucene inverted index (heavy index, fast read) vs Loki labels-only index + chunks (cheap write, grep-on-read)
execution-plan asymmetry: posting-list intersection vs label-resolve-then-stream-from-S3
cardinality: low-card → labels/fields, high-card → body / structured metadata
tiering: hot/warm/cold/frozen ILM and the two-orders-of-magnitude cost ladder
retention vs deletion: tombstones, force-merge lag, the PII per-template leak
head sampling vs tail sampling: when uniform sampling drops the only error
backend distribution: distributors, ingesters, queriers — partition by hash(doc_id) vs hash(label-set)