Scene 08

500 alerts, one page — alert grouping, inhibition and the shared notification log

Rules only decide that something is wrong; a separate alert router bundles alerts sharing labels into one notification, drops identical copies from redundant monitors, and holds back symptoms of a known cause, so a zone outage becomes one page.

Previously

Rules decide something is wrong; a separate system must decide who hears about it and how many times.

Scene 08

500 alerts, one page

Watch
Try it
Predict
Capture

What to watch for

What decides who gets told, how often, and what gets held back? Zone-b has just failed. Two monitors watch the same pods, so each of them fires a PodDown alert for all 500 — plus one ZoneDown alert naming the cause — and each sends every alert to both alert routers. Nothing is configured on the routers yet. Watch the pages-sent counter and let it finish.

Across the top: a banner naming the outage and whether the ZoneDown cause alert is firing ("500 pods down in zone-b · ZoneDown firing").
Left: two monitors watching the same pods, each sending every alert to BOTH routers — the crossed-out load balancer is the thing you must never put here.
Middle: the two alert routers, joined by a dashed shared log of what has already been sent, so the second one can see the first already notified.
Right: the router's two rules — bundling alerts that share an alert name and zone into one bucket that waits 30 s before its first notification (grouping), and a mute rule that holds a symptom back while its cause is firing (inhibition) — then the pages-sent counter on a 1-to-1,000 scale, ending at the on-call phone.

Implementation

Highlighted lines are the ones running in the diagram right now.

Monitor.send_alerts

every firing alert goes to every router, name tag stripped

1def send_alerts(firing):
2    for alert in firing:
3        # alert_relabel_configs drops the tag
4        #   naming which monitor sent it
5        if drop_monitor_tag:
6            del alert.labels["monitor"]
7        for router in all_routers:
8            # never behind a load balancer
9            router.post(alert)

Router.on_alert

each arriving alert lands in a bucket keyed by its labels

1def on_alert(alert):
2    if group_by:                 # alertname, zone
3        key = tuple(alert.labels[l] for l in group_by)
4    else:
5        key = alert.fingerprint  # ['...'] = no bundle
6    group = groups.setdefault(key, [])
7    group.append(alert)
8    if group.first_flush:
9        # window for a cause alert to arrive
10        schedule(flush_group, key, after=group_wait)
11    else:
12        schedule(flush_group, key, after=group_interval)

Router.flush_group

mute check first, then the shared log, then notify

1def flush_group(key):
2    alerts = groups.pop(key)
3    for rule in inhibit_rules:   # source -> target
4        if rule.source_is_firing and \
5           rule.equal_labels_match(alerts):
6            return               # held back, no notify
7    wait(self.position * 15)     # cluster.peer-timeout
8    if notification_log.has(key):
9        return                   # a peer already sent it
10    notify(receiver, alerts)
11    notification_log.add(key)

Sources

Where this sits in Metrics / Monitoring System

Scene 08 of 18, in the Wake a human act — Patient rules, one page, and a heartbeat for the monitor.. Rules only decide something is wrong: a separate router bundles alerts that share labels, drops identical copies from redundant monitors and holds back symptoms of a known cause, so a zone outage is one page.

Up next. Now that alerts reliably become one page, the next question is what happens when the pipeline delivering them is itself broken.

All 18 scenes in Metrics / Monitoring System · Every curriculum

Built with Arqly

Every scene in Metrics / Monitoring System builds on the one before it.All 18 Metrics / Monitoring System scenes