Scene 12

Split, shard, and cache the past

A query frontend splits a long-range query into day-sized pieces, spreads them across workers, merges them and caches finished days, which almost never change, so a refresh recomputes only the newest slice.

Previously

Years of data now sit in compacted day-sized blocks; a dashboard still has to read 30 days of them quickly.

Scene 12

Split, shard, and cache the past

Watch
Try it
Predict
Capture

What to watch for

How does a dashboard asking for 30 days of data come back fast? Right now it doesn't. Watch one panel's query — requests per second per route, over the last 30 days — arrive as a single job. One worker walks the whole range: the last few hours still live in the ingesters' memory, and everything older is read from the compacted day blocks you just built. The bar at the bottom is the wall-clock for the whole thing. Notice the box the query passes through on its way in: it is doing nothing yet, and it is the thing this scene turns on. Its name is the query frontend — the service that sits in front of the workers and is allowed to rewrite a query before anyone runs it.

Top left: the dashboard and the query it sends.
Beside it sits the box that query hits first — the query frontend, the service in front of the workers whose whole job is to rewrite one long query into smaller ones, hand them out and merge the answers.
The strip below is one cell per piece of the time range, oldest on the left, newest on the right; the coloured band under each cell says where that piece is read from (blue = compacted blocks in object storage, violet = the ingesters' memory, striped = the hour that both can serve).
Then the workers that actually run the pieces, and the results cache — a shelf of already-computed answers, one per piece, so a piece computed once can be handed back instead of run again.
Bottom: the wall-clock bar for the whole query, how many series it had to touch, and a rose banner if a limit refused it.

Implementation

Highlighted lines are the ones running in the diagram right now.

QueryFrontend.run

rewrites one long query into pieces and merges the answers

1def run(q, start, end):
2    if would_fetch_too_many_series(q, start, end):
3        refuse(max_fetched_series_per_query)
4    # split_queries_by_interval = 24h
5    pieces = split(align(start, end, q.step), 24h)
6    hits, misses = cache.get_or_miss(q, pieces)
7    fresh = workers.run_parallel(q, misses)
8    cache.put(q, fresh)
9    return merge(hits + fresh)

Cache.get_or_miss

decides which pieces the shelf is allowed to serve

1def get_or_miss(q, pieces):
2    hits, misses = [], []
3    for piece in pieces:
4        entry = None
5        # cache_results is off by default; freshness 10m
6        if cache_results and piece.end <= now() - 10m:
7            entry = memcached.get(key(q, piece, q.step))
8        if entry:
9            hits.append(entry.answer)
10        else:
11            misses.append(piece)
12    return hits, misses

Querier.execute

one worker runs one piece against ingesters and blocks

1def execute(q, piece):
2    series = []
3    for part in shard(q, n = 16):   # on by default
4        # query_ingesters_within = 13h
5        if piece.end > now() - 13h:
6            series += ingesters.select(part, piece)
7        # query_store_after = 12h, windows overlap
8        if piece.start < now() - 12h:
9            series += blocks.select(part, piece)
10    if chunks(series) > max_fetched_chunks_per_query:
11        abort("too many chunks")
12    return eval(q, series)

Sources

Where this sits in Metrics / Monitoring System

Scene 12 of 18, in the Scale out act — Remote-write, ingesters, dedup, blocks, split queries.. A query frontend splits a long range into day-sized pieces, spreads them across workers and caches finished days, which almost never change, so a refresh recomputes only the newest slice.

Up next. Now that query cost follows the number of series a query touches, the next question is why that number can explode even while the dashboard's count of live series stays flat.

All 18 scenes in Metrics / Monitoring System · Every curriculum

Built with Arqly

Every scene in Metrics / Monitoring System builds on the one before it.All 18 Metrics / Monitoring System scenes