Scene 08a

Silence looks exactly like health — the dead man's switch heartbeat alert

When the monitoring pipeline dies no alerts fire, which looks exactly like health, so you keep an always-firing heartbeat alert that an outside service expects to hear and pages when it stops.

Previously

Alerts reliably become one page, but only while every stage of the pipeline is alive.

Scene 08a

Silence looks exactly like health

Watch
Try it
Predict
Capture

What to watch for

How do you find out that the monitoring itself is broken? Everything you have learned so far assumes the pipeline is running. Watch one Friday night: the monitor's memory bar climbs, it dies at 02:00, and five minutes later checkout starts failing. Keep your eyes on the two panels at the bottom — the alerts dashboard and the on-call phone — and ask yourself how this night differs from a night where nothing happened at all.

Top right is the time-to-detect clock: how long checkout stays broken before a human is woken.
Along the top is the pipeline strip — instrument → collect → store → query → alert → notify — with a break marked on whatever is broken right now.
Inside the dashed YOUR CLUSTER box are the three stages that carry a page to a human: the monitor, the alert router, and the pager integration.
Under the monitor is its memory bar with the rose customer_id chip from the Friday deploy; beside it, the night's timeline.
The violet dotted line is the heartbeat alert: an alerting rule whose condition is always true, so it is always firing, sent down the same path a real page takes.
It ends at the outside check — a separate service at a different provider that expects the pulse and raises its own page when the pulse stops arriving.
A check wired backwards like that, where absence is the alarm, is a dead man's switch, and watching your monitoring this way is meta-monitoring.
Bottom left are the only two things the on-call engineer actually has: the alerts dashboard and the phone.

Implementation

Highlighted lines are the ones running in the diagram right now.

Rule.watchdog

an alerting rule whose condition is always true

1watchdog = Rule(
2    expr = "vector(1)",        # always true
3    route_to = "outside-check",
4)
5def evaluate(rules):   # every evaluation_interval
6    for rule in rules:
7        if rule.expr_is_true():
8            a = rule.fire()
9            a.valid_until = now() + 4 * max(
10                evaluation_interval, resend_delay)
11            router.post(a)
12# all of this runs inside the monitor process

Router.notify

delivers firing alerts, resolves the ones that stop renewing

1def on_alert(alert):          # posted by the monitor
2    groups[alert.key].append(alert)
3    schedule(flush, alert.key, after=group_wait)
4def flush(key):
5    for alert in groups.pop(key):
6        if now() > alert.valid_until:
7            resolve(alert)    # no renewal arrived
8            continue
9        receiver_of(alert).send(alert)
10        # the watchdog's receiver is the outside check
11# both halves run inside your cluster

OutsideCheck.on_tick

a timer at another provider, watching for the pulse

1last_seen = None       # state at a different provider
2def on_pulse(alert):
3    if alert.is_firing:
4        last_seen = now()
5def on_tick():
6    if not configured:
7        return
8    if now() - last_seen <= timeout:  # 5 min (example)
9        return
10    page(on_call, path=outside_the_cluster)

Sources

Where this sits in Metrics / Monitoring System

Scene 08a of 18, in the Wake a human act — Patient rules, one page, and a heartbeat for the monitor.. When the monitoring pipeline dies no alerts fire, which looks exactly like health — so an always-firing heartbeat leaves the pipeline and an outside service pages when it stops arriving.

Up next. Now that we can tell when the monitor dies, the next question is the most common reason it dies at scale: one server can't hold every series.

All 18 scenes in Metrics / Monitoring System · Every curriculum

Built with Arqly

Every scene in Metrics / Monitoring System builds on the one before it.All 18 Metrics / Monitoring System scenes