Hot, warm, cold, frozen — log storage tiers and ILM phases

Logs grow forever and nobody can afford 'all logs on NVMe forever' — the economics of two orders of magnitude between NVMe and object storage force log data through tiers. ELK has four ILM phases (HOT → WARM → COLD → FROZEN) that read from progressively cheaper storage; Loki collapses the layers because chunks live on S3 from day one. ILM only DEMOTES bytes here — when bytes DIE is scene 8.

Previously

High-cardinality data has to live in the body, and the body is BIG and most of it is OLD. NVMe is too expensive to hold months of body — economics force tiering.

Scene 07

Hot, warm, cold, frozen

Watch
Try it
Predict
Capture

Day 0 — HOT (NVMe, ~$0.10/GB-mo)

What to watch for

Watch one index tile age. It's born today on HOT (NVMe, replicated, ms latency). The ILM clock advances 7 days — it slides to WARM (force-merged, fewer replicas). 30 days — it slides to COLD (a searchable snapshot in S3, fully mounted, ~50% disk savings, no replicas). 90 days — it slides to FROZEN (small NVMe cache, partially mounted from S3, up to 20× warm capacity). To the right, the Loki lane shows the same body parked on S3-Standard from day 0 — no movement, ever.

Implementation

Highlighted lines are the ones running in the diagram right now.

ILM.tick

the daily clock — for each index, evaluate phase actions

1# runs once per day on the master node
2def ilm_tick():
3    for index in cluster.indices:
4        age = now() - index.creation_date
5        policy = index.ilm_policy
6        if age >= policy.hot.max_age:      # rollover
7            rollover(index)
8        if age >= policy.warm.min_age:
9            demote(index, phase='warm')
10        if age >= policy.cold.min_age:
11            demote(index, phase='cold')
12        if age >= policy.frozen.min_age:
13            demote(index, phase='frozen')

ILM.demote

the phase transition — force_merge, snapshot, partial mount

1def demote(index, phase):
2    if phase == 'warm':
3        force_merge(index, max_num_segments=1)
4        set_replicas(index, 0)
5        allocation.require(data_warm)
6    elif phase == 'cold':
7        snap = searchable_snapshot(index, repo=s3)
8        mount(snap, type='full_copy')   # ~50% disk savings
9        delete_local_index(index)
10    elif phase == 'frozen':
11        snap = searchable_snapshot(index, repo=s3)
12        mount(snap, type='partial')     # NVMe cache + S3

Loki.compact_index

Loki's 'one tier' reality — only the index gets compacted

1# chunks were written to S3 the moment they flushed —
2# the body never moves. Only the index is compacted.
3def compact_index():
4    shards = list_index_shards(object_store)
5    for day in shards.by_day():
6        merged = merge_boltdb_shards(day)  # → TSDB
7        upload(merged, object_store)
8        delete(day.original_shards)
9    # chunks: untouched, still on S3-Standard from day 0

Where this sits in Build a distributed logging stack (ELK / Loki)

Scene 07 of 12. Two orders of magnitude in cost between NVMe and Deep Archive force tiering. ELK has four ILM phases; Loki collapses to S3 from day one.

Up next. We have a cost ladder from NVMe to Deep Archive — two orders of magnitude. Aging data through it is the policy engine; but the policy decides when bytes MOVE, not when bytes DIE.

All 12 scenes in Build a distributed logging stack (ELK / Loki) · Every curriculum

Built with Arqly

Every scene in Build a distributed logging stack (ELK / Loki) builds on the one before it.All 12 Build a distributed logging stack (ELK / Loki) scenes