Scene 04

Pull or push: who starts the conversation — scrape targets, the up metric and service discovery

When the monitor pulls each target on its own schedule, a failed fetch itself says the target is down, while when apps push, short-lived jobs can still report but silence becomes ambiguous: dead, or just quiet?

Previously

Totals survive missed reads; now we decide who starts each read and what a failed read tells us.

Scene 04

Pull or push: who starts the conversation

Watch
Try it
Predict
Capture

What to watch for

How do the numbers get from the app to the monitor, and what does each direction cost? Here the monitor does the asking. It first gets Kubernetes' list of checkout pods that exist right now (finding targets this way is called service discovery), then walks that list, fetching each pod's page of numbers every 15 s (example). Watch the second round: pod-7 hangs while Kubernetes still lists it.

Top: the pipeline strip, measure → collect → keep → query → check → notify, with collect opened up below.
Left: the Kubernetes pod list (who exists right now) feeding the monitor.
Right: ten checkout pods, each with a /metrics page of running totals.
A scrape is one fetch of a pod's /metrics page by the monitor; each scrape also writes up = 1 if it worked, 0 if it failed.
A push is the reverse: the app sends its own numbers, so there's no fetch to fail.
In relay position, a relay box keeps each sender's last value and is fetched instead.
Bottom: a batch job's run drawn against the fetch ticks.
Top-right: time-to-detect (TTD), example values.

Implementation

Highlighted lines are the ones running in the diagram right now.

Monitor.scrape

the monitor asks every listed target for its page

1def scrape_all():  # every 15 s (example)
2    targets = kubernetes.list_pods("checkout")
3    for target in targets:
4        page = http_get(target.url + "/metrics")
5        if page is None:
6            store.write("up", target, 0)
7            continue
8        store.write("up", target, 1)
9        for series, value in parse(page):
10            store.write(series, target, value)

App.push

each pod sends its own totals when it chooses

1def push_loop(target):  # the monitor, or a relay
2    while True:
3        totals = read_totals()
4        target.send(self.name, totals)
5        sleep(push_interval)  # 15 s (example)

Relay.serve

keeps what senders sent; answers the monitor's fetch

1last = {}
2def on_push(sender, totals):
3    last[sender] = totals
4def on_scrape():
5    return [(sender, totals)
6            for sender, totals in last.items()]

BatchJob.run

a job that exists only while it works

1def run_job(target):
2    serve("/metrics")  # open only while running
3    result = do_work()  # 3 s, 30 s or 5 min
4    if target is not None:
5        target.send("batch-job", result)
6    exit()

Sources

Where this sits in Metrics / Monitoring System

Scene 04 of 18, in the Collect act — Running totals, and who starts the conversation.. When the monitor pulls each target on its own schedule, a failed fetch itself says the target is down; when apps push, silence becomes ambiguous — dead, or just quiet?

Up next. Now that the monitor reliably collects every pod's running totals, the next question is how to turn ever-growing totals from many pods into one honest requests-per-second line.

All 18 scenes in Metrics / Monitoring System · Every curriculum

Built with Arqly

Every scene in Metrics / Monitoring System builds on the one before it.All 18 Metrics / Monitoring System scenes