Scene 01

Find out before your customers do — the monitoring pipeline and time-to-detect

A monitoring system turns 'what is our software doing right now?' into numbers measured on a fixed schedule, kept in one place and checked by rules on a fixed schedule, so a machine rather than a customer is first to notice checkout is broken.

Scene 01

Find out before your customers do

Watch
Try it
Predict
Capture

What to watch for

What does it take to find out your site is broken before your customers tell you? At 02:14 Shopfront's checkout servers start answering requests with errors. Every failed request is written to a log, and nobody is counting anything. Watch how long it takes before anyone notices. Then the same night replays, this time with the servers counting their errors every 15 s and a rule checking that count every 15 s (an example setup). Watch which lane lights first, and the time-to-detect readout that appears top-right.

Top: the monitoring pipeline, measure → collect → keep → query → check → notify; this scene lights only measure, keep and check, the rest come later.
Middle: three ways of finding out checkout broke at 02:14 (complaints, a glance at a dashboard, a rule that checks every 15 s), each marking when it first notices.
Bottom: checkout's error count.
A metric is one number about the software measured on a schedule, like 'checkout errors in the last 15 s'.
A time series is that metric's row of values over time, one every 15 s.
Top-right: time-to-detect (TTD), the time from checkout breaking to a human being paged (example values).

Implementation

Highlighted lines are the ones running in the diagram right now.

App.handle_checkout

each checkout request; failures logged and counted

1def handle_checkout(request):
2    result = checkout(request)
3    if result.failed:
4        log.write(request, result.error)
5        checkout_errors += 1
6    return result

App.measure

turns the error count into one number per interval

1every 15 s:  # measure interval (example)
2    value = checkout_errors
3    checkout_errors = 0
4    store.append("checkout_errors", now(), value)

Dashboard.watch

a person looking at the numbers on a screen

1def watch_dashboard():
2    while person.at_screen():
3        screen.draw(store.series("checkout_errors"))
4        if person.glances() and screen.shows_spike():
5            person.page(on_call)

Rule.check

a machine compares the latest number to a limit

1every 15 s:  # check interval (example)
2    latest = store.last("checkout_errors")
3    if latest > limit:
4        pager.page(on_call, "checkout is failing")

Sources

Where this sits in Metrics / Monitoring System

Scene 01 of 18, in the Why measure act — Numbers beat complaints; labels, not traffic, set the bill.. Monitoring means numbers measured on a schedule and checked by rules on a schedule, so a machine — not a customer — is first to notice checkout is broken.

Up next. Now that a machine checks numbers on a schedule, the next question is what keeping those numbers costs, and what makes them expensive.

All 18 scenes in Metrics / Monitoring System · Every curriculum

Built with Arqly

Every scene in Metrics / Monitoring System builds on the one before it.All 18 Metrics / Monitoring System scenes