Scene 16

Design your monitoring platform

A monitoring platform is a chain of decisions about collection, alerting, scaling out, retention and label budget, each right only for a workload whose series count, team count, retention and detection target justify it.

Previously

You can now control what every series costs without losing answers; the last step is choosing which of these pieces a given workload actually needs.

Scene 16

Design your monitoring platform

Watch
Try it
Predict
Capture

What to watch for

Can you put every piece together for a real workload and defend each choice? Here is the workload: 30 teams, 50k pods, 13 months of history, and a page within 2 minutes of checkout breaking. The card's blue numbers are estimates derived from Shopfront's label math — roughly 300 series per pod, so ≈ 15M active series, and at 3 copies in memory that is ≈ 375 GB of ingester RAM. Grafana's own advice is to measure your own cluster rather than trust an estimate. The pipeline strip below is empty: nothing has been placed yet. Watch the checker grade that empty design.

Top-left: three workload tabs.
Top-right: the time-to-detect clock with this workload's target.
The card under the tabs holds the workload's facts and, in dashed blue, the estimates derived from Shopfront's label math (≈ 300 series per pod) — estimates, not measurements.
Below it runs the pipeline strip, instrument → collect → store → query → alert → notify, with the design's pieces stacked under the stage they belong to: solid blue where the preset chose them, faint outlines where it didn't, and the rose customer_id chip crossed out when it is kept out.
The amber row underneath fires all three failures at once and marks which piece held.
On the right, the checker: one row per requirement, ✓ / ~ / ✗, and under each row the scene that explains the rule.
The pill at the bottom right is the verdict.

Implementation

Highlighted lines are the ones running in the diagram right now.

Checker.size_the_fleet

every number it returns is an estimate, not a measurement

1def size_the_fleet(workload, design):
2    # estimate from Shopfront's label math
3    active = workload.pods * 300
4    if design.customer_id_label:
5        active *= workload.customers   # 2M values
6    if not design.cluster:
7        return active <= one_server_ceiling  # ~10M
8    in_memory = active * copies        # copies = 3
9    ram_gb = in_memory / 300_000 * 2.5 # estimate
10    return active, ram_gb

Checker.time_to_detect

the clock is a sum of four waits, checked against the target

1def time_to_detect(workload, design):
2    if not design.rules:
3        return "never"             # nothing evaluates
4    ttd = scrape_interval           # 15 s
5    ttd += evaluation_interval      # 15 s
6    ttd += design.for_duration      # 1 min, or 2 min
7    ttd += group_wait               # 30 s bundling
8    for f in design.failures:
9        if absorbed(f):  # twin alerts, 2 of 3 confirm
10            continue     # the clock does not move
11        return "never"  # the monitor itself is gone
12    return ttd <= workload.target

Checker.verify

one row per rule this workload asks for, each naming its scene

1RULES = [                        # each row cites a scene
2    ("fits memory",     fits_memory,   "metrics-09"),
3    ("series budget",   under_budget,  "metrics-02"),
4    ("13 months kept",  has_compactor, "metrics-11"),
5    ("which customer?", has_exemplars, "metrics-15"),
6    ("complexity",      is_justified,  "metrics-09")]
7def verify(workload, design):
8    rows = []
9    for check, rule, scene in RULES:
10        if rule.applies(workload):
11            rows += row(check, rule(workload, design), scene)
12    return rows

Sources

Where this sits in Metrics / Monitoring System

Scene 16 of 18, in the Cardinality act — Churn, defenses, aggregation, then design it.. A monitoring platform is a chain of decisions — collection, alerting, scaling out, retention and label budget — each right only for a workload whose series count, teams, retention and detection target justify it.

All 18 scenes in Metrics / Monitoring System · Every curriculum

Built with Arqly

Every scene in Metrics / Monitoring System builds on the one before it.All 18 Metrics / Monitoring System scenes