Scene 11

Keep a year cheaply: blocks on object storage

Ingesters hold only the recent hours in memory and every two hours seal that window into an immutable block on object storage, where a compactor merges the three copies replication created and joins small blocks into day-sized ones — which turns long retention from a memory problem into a storage-cost and query-speed problem.

Previously

Now that exactly one stream per series lands in memory, the next question is how the cluster keeps months of it without holding it all in memory.

Scene 11

Keep a year cheaply: blocks on object storage

Watch
Try it
Predict
Capture

What to watch for

How does a metrics cluster keep months of data without holding it in memory? Watch one tenant's two hours travel. An ingester keeps only recent samples in memory, so every two hours it seals that window into a file and uploads it to object storage — the cheap, durable, write-once store build-s3 covered (s3-00). Then watch what the other two ingesters do with the same two hours.

Top left: three ingesters, each holding the most recent ~2 h of a tenant's samples in memory; the amber chip on one means it is sealing that window into a file and uploading it.
Top right: the compactor, the background job that rewrites what is already in object storage.
Middle: the object-storage lane, one tile per 2 h window, oldest on the left; each tile is a block — one sealed file holding that window's samples, written once and never edited again.
An amber stack marked ×3 is the same window uploaded by three different ingesters, a plain blue tile is a single copy, and a wide green tile is several windows joined into one larger file (the number inside is the span it covers).
Thin violet strips under a tile are extra lower-resolution copies of it.
Bottom left: retention, with the amber bar for memory plus local disk and the blue bar for object storage.
Bottom right: the storage multiple, what a 30-day query has to open, and the downsampling switch.

Implementation

Highlighted lines are the ones running in the diagram right now.

Ingester.cut_block

seal the last two hours, upload it, keep it locally a while

1def cut_block(tenant):
2    block = seal(tenant.head)     # never edited again
3    upload(bucket, block)
4    keep_local(block, for_="13h")
5    tenant.head = fresh_head()
6 
7every("2h"):
8    for ingester in owners_of(tenant.series):
9        ingester.cut_block(tenant)

Compactor.run

the background job that rewrites what is already stored

1def run():
2    if not compactor_enabled:
3        return                    # nothing is rewritten
4    for window in bucket.windows():
5        copies = bucket.blocks(window)
6        if len(copies) > 1:
7            merge_vertical(copies)
8    for group in adjacent(bucket):
9        merge_horizontal(group)   # 2h -> 12h -> 24h
10    for block in bucket.blocks():
11        if block.age > blocks_retention_period:
12            delete(block)         # unset = kept forever

Compactor.downsample

coarser copies of old blocks, kept beside the raw one

1def downsample(block):
2    if not downsampling_enabled:
3        return                   # Mimir removed this path
4    if block.age > "40h":
5        coarse = aggregate(block, every="5m")
6        upload(bucket, coarse)   # added, not swapped
7    if block.age > "10d":
8        coarse = aggregate(block, every="1h")
9        upload(bucket, coarse)   # added, not swapped

Sources

Where this sits in Metrics / Monitoring System

Scene 11 of 18, in the Scale out act — Remote-write, ingesters, dedup, blocks, split queries.. Ingesters keep only recent hours in memory and cut immutable two-hour blocks to object storage, where a compactor merges the copies replication created — retention becomes a storage and query-speed problem.

Up next. Now that years of data sit in compacted blocks, the next question is how a 30-day query over them comes back fast.

All 18 scenes in Metrics / Monitoring System · Every curriculum

Built with Arqly

Every scene in Metrics / Monitoring System builds on the one before it.All 18 Metrics / Monitoring System scenes