Scene 09

One monitor hits a wall — splitting targets, federation and remote-write

One server's memory grows with its series, so a growing fleet has three exits, each giving something up: split targets so fleet-wide questions must ask every server, federate pre-summed series and lose per-pod detail, or remote-write every sample to a cluster you must now run.

Previously

Now that an outside heartbeat tells us when the monitor dies, here is the most common reason it dies at scale: its memory must hold every series, and the fleet keeps growing.

Scene 09

One monitor hits a wall

Watch
Try it
Predict
Capture

the bar is series held in memory — it tracks how many pods exist, not how busy they are

What to watch for

What are your options when one monitoring server can't hold every series? Watch the left side. Shopfront's fleet grows from 1k pods to 50k, and the monitor's memory bar climbs with it, because a monitor keeps every series it is currently collecting in memory — build-tsdb showed why (tsdb-09). At the top of the slider the bar breaks through the budget line of the biggest machine you can rent. This is the wall build-tsdb handed over as somebody else's problem (tsdb-10a). Nothing is fixed yet; just watch the bar.

Left: the pod fleet (blue dots) feeding one monitor.
The bar is the memory its series occupy; the dashed line is the RAM budget of the machine it runs on, and a rose ✕ OOM burst caps the bar once it breaks through that line.
Middle: the way out you picked — several monitors side by side, and the box above them that ties them together.
That box is either a query layer that asks every server and merges the replies, a global server fed by federation (one server copies a chosen set of series from each monitor, so a single view spans them all), or the grey cluster that remote-write feeds (the monitor pushes every sample it collects out to another system as the sample arrives, instead of keeping it).
The amber chips under that middle column are the price of the way out you picked.
Right: the question card — one fleet-wide question, and whether each way out can still answer it.

Implementation

Highlighted lines are the ones running in the diagram right now.

Shard.assign_targets

each server scrapes one bucket; fleet questions ask them all

1def assign_targets(all_pods, shard_id):
2    n_shards = ceil(series_of(all_pods) / ram_budget)
3    mine = []
4    for pod in all_pods:        # from service discovery
5        if hash(pod.labels) % n_shards == shard_id:
6            mine.append(pod)    # hashmod relabeling
7    return mine
8 
9def fleet_wide(query):
10    replies = [srv.ask(query) for srv in servers]
11    return merge(replies)       # one round trip each

GlobalServer.federate

pulls /federate on an interval and stores what comes back

1def federate():
2    for leaf in regional_servers:
3        resp = leaf.get("/federate", match=RULE_SUMS)
4        for series in resp:
5            store(series.labels, series.value)
6    sleep(scrape_interval)
7 
8def answer(query):
9    return evaluate(query, over=store)

Sender.remote_write

streams the write-ahead log out, batch by batch

1def remote_write():
2    while True:
3        batch = wal.read(max_samples_per_send)
4        shard = queue.pick(max_shards)   # ~25% more RAM
5        if shard.send(batch, to=receiver):
6            wal.advance(len(batch))
7        else:
8            shard.retry(batch)
9        if wal.oldest() > 2h:
10            wal.truncate()   # unsent data lost after ~2 h

Sources

Where this sits in Metrics / Monitoring System

Scene 09 of 18, in the Scale out act — Remote-write, ingesters, dedup, blocks, split queries.. One server's memory holds every active series, so a growing fleet has three exits: split targets and query them all, federate sums and lose per-pod detail, or remote-write to a cluster you must now run.

Up next. Now that remote-write sends every sample to a cluster, the next question is what that cluster does with a sample so no single machine is overloaded — or loses it.

All 18 scenes in Metrics / Monitoring System · Every curriculum

Built with Arqly

Every scene in Metrics / Monitoring System builds on the one before it.All 18 Metrics / Monitoring System scenes