Cardinality is the killer

Build a distributed logging stack (ELK / Loki) (12 scenes)

Scene 06 · Cardinality is the killer

Every unique label-set is a Loki stream; every dynamic key is an ELK mapping field. Putting request_id in either kills the index in minutes — low-card → labels, high-card → body.

Previously

ELK answered the query in milliseconds because the index already knew which documents contained `ERROR`. The temptation is to index more things — but 'more things' has unbounded values, and unbounded values destroy the index on both systems.

Scene 06

Cardinality is the killer

Watch

Diagram

A two-pane Y-fork from one 'add a label / add a field' control. LEFT pane (Loki): a stream-tile grid sized service(3) × env(2) × level(3) = 18 cells at baseline, with a label-picker dropdown. Adding a dimension MULTIPLIES the grid; counters track total streams against a 100k tenant ceiling, plus a chunks-on-S3 swarm and an index-size meter that pierces a red line under request_id. RIGHT pane (ELK): a mapping panel with a 47/1000 field-budget gauge and a dynamic-key emitter — emitting JSON whose KEYS vary walks the gauge past 1,000 and triggers a 'cluster-state oversize' banner. A central rule-card at the bottom reads: low-cardinality → labels (Loki) / mapped fields (ELK); high-cardinality → body / structured metadata. The card tints amber while a side is exploding and green once the offending dimension is moved to the body.

Both backends start at green baselines. LEFT pane (Loki) has 18 streams from service × env × level. RIGHT pane (ELK) has 47 of 1,000 mapped fields used. Read the rule-card at the bottom but don't act on it yet — Manipulate is where you'll feel why it matters.

Implementation

Loki.ensure_stream

every unique label-set is its own stream + chunk

1def ensure_stream(tenant, label_set):
2    stream_id = hash(sorted(label_set.items()))
3    if stream_id in active_streams[tenant]:
4        return active_streams[tenant][stream_id]
5    # new label-set -> new stream, new chunk, new index entry
6    if len(active_streams[tenant]) >= max_streams_per_user:
7        reject('per-stream limit exceeded')
8    chunk = open_chunk(stream_id)
9    active_streams[tenant][stream_id] = chunk
10    streams_created_total.inc()
11    return chunk

ELK.dynamic_mapping

each new JSON key becomes a mapped field

1def index_document(index, doc):
2    for key, value in flatten(doc).items():
3        if key not in mapping[index].fields:
4            if mapping[index].dynamic is False:
5                continue  # ignored, not indexed
6            if len(mapping[index].fields) >= 1000:
7                # index.mapping.total_fields.limit
8                raise MappingExplosion(index)
9            mapping[index].fields[key] = infer_type(value)
10        write_to_inverted_index(index, key, value)

Pipeline.relabel

the rule-card fix: high-card out of the indexed shape

1def relabel(entry):
2    # Low-cardinality dimensions stay as labels / mapped fields.
3    indexed = pick(entry, ['service', 'env', 'level', 'region'])
4    # High-cardinality dimensions move to the body /
5    # structured metadata, searchable by line filter only.
6    body = entry.body
7    body['request_id'] = entry.pop('request_id', None)
8    body['user_id']    = entry.pop('user_id', None)
9    body['trace_id']   = entry.pop('trace_id', None)
10    return { 'labels': indexed, 'body': body }

PreviousSame query, two execution plans NextHot, warm, cold, frozen