Scene 07

An alert is a query with patience — the for duration, pending and firing states

An alerting rule is a query re-run at a fixed interval whose for duration holds the alert pending until the condition has stayed true continuously that long, trading a known extra delay for far fewer false pages.

Previously

We can compute honest numbers; now a machine must decide when one of them is bad enough, and long enough, to wake someone.

Scene 07

An alert is a query with patience

Watch
Try it
Predict
Capture

What to watch for

How does a number become a page without paging on every blip? A machine has to watch checkout's 5xx ratio for us. Watch the ticks march along the bottom: at every one of them the same query runs again and the answer is compared to the threshold. The strip underneath records what the alert decided after each evaluation. Two brief spikes and one real outage go past — see which of them actually rings a bell.

The blue line is checkout's 5xx ratio, one point per evaluation tick along the bottom (15 s apart in this example).
The dashed red line is the threshold the rule compares against, and the header states the rule itself: a query re-run on a schedule, plus how long the answer must stay bad — that whole thing is an alerting rule.
The strip under the plot is the alert's state after each evaluation: grey inactive, amber pending (the condition is true but has not held long enough yet), red firing (it held, so a page goes out).
A ↺ mark under the strip is drawn wherever an amber run ended without firing — that is the clock being thrown away.
The red-shaded band between the two dashed verticals is the stretch where checkout genuinely is broken.
The small dial is the waiting clock filling up on the current amber run; it empties whenever an evaluation comes back false.
Bells on the top row mark pages actually sent, the counter on the right estimates pages per week, and the top-right clock is time-to-detect.

Implementation

Highlighted lines are the ones running in the diagram right now.

Rule.evaluate

one query, one comparison, re-run on a timer

1def evaluate(rule):  # every evaluation_interval
2    value = query(rule.expr)
3    if value > rule.threshold:
4        update_state(rule, now())
5    else:
6        clear(rule)
7 
8def run_rules(rules):
9    while True:
10        for rule in rules:
11            evaluate(rule)
12        sleep(evaluation_interval)  # 1m default

Rule.update_state

pending while the run is short, firing once it is long enough

1def update_state(rule, now):
2    if rule.active is None:
3        rule.active = Alert(started_at=now)
4    held = now - rule.active.started_at
5    if held >= rule.for_duration:  # `for`
6        rule.active.state = FIRING
7        page(rule.active)
8    else:
9        rule.active.state = PENDING
10 
11def clear(rule):
12    del rule.active

Pager.page

only a firing alert sends, and only once per run

1def page(alert):
2    if alert.state != FIRING:
3        return
4    if alert.already_paged:
5        return
6    alert.already_paged = True
7    send_page(alert)
8 
9def worst_case_ttd(rule):
10    return (scrape_interval
11            + evaluation_interval
12            + rule.for_duration)

Sources

Where this sits in Metrics / Monitoring System

Scene 07 of 18, in the Wake a human act — Patient rules, one page, and a heartbeat for the monitor.. An alerting rule is a query re-run every interval; its waiting period holds the alert pending until the condition has stayed true that long, trading a known delay for far fewer false pages.

Up next. Now that rules decide something is wrong, the next question is who gets told, and how 500 firing alerts avoid becoming 500 pages.

All 18 scenes in Metrics / Monitoring System · Every curriculum

Built with Arqly

Every scene in Metrics / Monitoring System builds on the one before it.All 18 Metrics / Monitoring System scenes