Scene 10

Hash every series to three ingesters

The cluster's front door hashes each series' full label set onto a ring and writes every sample to the same three ingesters, succeeding once two confirm, so losing one ingester loses nothing and no popular metric lands on one machine — at the price of every query asking all ingesters.

Previously

Now that remote-write sends every sample to a cluster, the next question is what the cluster does with a sample so no single machine is overloaded or lost.

Scene 10

Hash every series to three ingesters

Watch
Try it
Predict
Capture

What to watch for

What happens to a sample when it enters a scalable metrics cluster? Remote-write hands the cluster a batch, and something has to decide which machine keeps it. Watch one series go through: the front door checks it against the limits, hashes it onto the ring to pick its machines, and sends it to three of them. Two confirmations are enough to call the write done — the third arrives a moment later.

Across the top is the six-stage pipeline strip, with store expanded to "cluster store" — this whole diagram is the inside of that one box.
Top-left: one incoming remote-write batch carrying a single series.
The blue box is the distributor — the cluster's front door: it checks each series against the limits (too many labels, labels too long, timestamps too old, a tenant — one team's slice of a shared cluster — sending too fast → rejected with an HTTP error), then hashes the series onto the ring to decide where it goes.
The dashed circle is that hash ring (the same placement idea build-wcs taught in wcs-04); the six boxes on it are ingesters — the machines that hold the most recent samples in memory before they are written out for the long term.
Each ingester shows a memory bar.
Solid animated arrows are the write going to its owning ingesters; a green ✓ is that ingester confirming.
The pill under the distributor is the write's verdict.
On the right, the query box's faint dashed lines show how many ingesters a read has to ask.
At the right of the batch row is the current shard key and the copies-per-series setting with its memory cost.
Top-right is time-to-detect: how long checkout can be broken before a human is paged.

Implementation

Highlighted lines are the ones running in the diagram right now.

Distributor.push

check each series against the limits, then hash it

1def push(tenant, batch):
2    for series in batch:
3        if bad_labels(series) or too_old(series):
4            reject(series, 400)      # never stored
5        elif over_rate_limit(tenant):
6            reject(series, 429)      # tenant too fast
7        else:
8            if shard_by_metric_name:
9                key = fnv32a(tenant, series.name)
10            else:
11                key = fnv32a(tenant, series.labels)
12            replicate(key, series)

Distributor.replicate

send to every owner, answer once the quorum confirms

1def replicate(key, series):
2    owners = ring.owners(key, n=copies)
3    quorum = 2 if copies == 3 else 1
4    acks, fails = 0, 0
5    for ingester in send_parallel(owners, series):
6        if ingester.confirmed:
7            acks += 1
8        else:
9            fails += 1           # dead or too slow
10        if acks >= quorum:
11            return OK            # a third can be late
12    return ERROR                 # 500 to remote-write

Querier.select

a read asks whichever ingesters could hold a match

1def select(tenant, matchers):
2    if shard_by_metric_name and has_name(matchers):
3        key = fnv32a(tenant, name_of(matchers))
4        targets = ring.owners(key, n=copies)
5    else:
6        targets = all_ingesters
7    samples = []
8    for ingester in send_parallel(targets, matchers):
9        samples += ingester.recent_samples(matchers)
10    return dedupe(samples)

Sources

Where this sits in Metrics / Monitoring System

Scene 10 of 18, in the Scale out act — Remote-write, ingesters, dedup, blocks, split queries.. The cluster's front door hashes each series' full label set onto a ring and writes to three ingesters, succeeding once two confirm — no popular metric lands on one machine, but every query asks them all.

Up next. Now that the cluster keeps three safe copies of whatever it receives, the next question is what happens when the scrapers themselves are redundant and send every sample twice.

All 18 scenes in Metrics / Monitoring System · Every curriculum

Built with Arqly

Every scene in Metrics / Monitoring System builds on the one before it.All 18 Metrics / Monitoring System scenes