Scene 03

Running totals survive missed reads — counters, gauges and reset-on-read

A counter reports a running total that only goes up and a gauge reports a current level, and because totals are reported instead of counts since the last read, a missed or doubled read loses at most detail.

Previously

Each series is one number reported every interval; now we decide what that number means so lost or repeated reads can't corrupt it.

Scene 03

Running totals survive missed reads

Watch
Try it
Predict
Capture

What to watch for

What shapes can a metric take, and why report 'total so far' instead of 'since last time'? Checkout holds a number of requests served, and a collector reads it every 15 seconds (example). Watch the number climb from read to read, then look at the panel below: in-flight requests, the requests being handled right now, rise and fall with load.

Top: checkout pod's requests number, one point per read every 15 s (example); the header says whether the pod reports a running total or resets to 0 on every read.
Middle: collector A's and collector B's reads (B reads 1 s after A); a hollow dashed 'miss' tick is a read whose reply was lost.
Right: what each collector recorded versus the true request count.
Bottom: in-flight requests, a number that rises and falls.
A counter is a running total that only goes up and returns to 0 only when the process restarts.
A gauge is a current level, like in-flight requests, that can go up or down.
Top-right: time-to-detect (example).

Implementation

Highlighted lines are the ones running in the diagram right now.

App.handle_request

one number that only climbs, one that rises and falls

1def handle_request(req):
2    in_flight += 1
3    requests_served += 1
4    since_last_read += 1
5    resp = checkout(req)
6    in_flight -= 1
7    return resp

App.on_read

what the pod hands back, in each report style

1def on_read():
2    if report_running_total:
3        return requests_served
4    count = since_last_read
5    since_last_read = 0
6    return count

Collector.read

each collector asks the pod once per interval

1def read(collector):  # every 15 s (example)
2    reply = pod.on_read()
3    if reply is None:
4        return
5    collector.recorded.append(
6        Sample(now(), reply),
7    )

Collector.counted

turning stored reads into a request count

1def counted(collector):
2    reads = collector.recorded
3    if report_running_total:
4        return reads[-1].value - reads[0].value
5    return sum(r.value for r in reads)

Sources

Where this sits in Metrics / Monitoring System

Scene 03 of 18, in the Collect act — Running totals, and who starts the conversation.. A counter reports a running total that only goes up and a gauge a current level; because totals are reported, a missed or doubled read loses at most detail, never the count itself.

Up next. Now that running totals survive missed and repeated reads, the next question is who starts each read: does the monitor fetch the numbers, or does the app send them?

All 18 scenes in Metrics / Monitoring System · Every curriculum

Built with Arqly

Every scene in Metrics / Monitoring System builds on the one before it.All 18 Metrics / Monitoring System scenes