An agent on every host — at-least-once log shipping and offset checkpoints

A shipping agent on each host owns three pieces of state — a byte offset on disk (positions.yaml / registry), an in-RAM batch, and a retry slot — and ships batches at-least-once, which means duplicates on retry are normal and a checkpoint that advances before the ship can silently lose lines.

Previously

Centralising means putting a network between the app and its log file — and that something on each host that has to keep the pipe full has a name. It is the shipping agent: Filebeat, Promtail, Fluent Bit, or Vector. They look different on the outside; under the skin they are the same three compartments.

Scene 02

An agent on every host

Watch
Try it
Predict
Capture

What to watch for

The app writes lines into /var/log/app.log. The agent reads each new line into its in-memory batch. When the batch hits 4 lines, it POSTs the batch over HTTP. Only AFTER the backend acks does positions.yaml advance — that on-disk offset is what the agent re-reads from on restart.

Implementation

Highlighted lines are the ones running in the diagram right now.

Agent.tail_loop

reads new lines from app.log into the in-RAM batch

1def tail_loop():
2    f = open('/var/log/app.log')
3    f.seek(self.offset)  # offset from positions.yaml
4    while running:
5        line = f.readline()
6        if not line:
7            sleep(poll_interval); continue
8        batch.append(line)
9        if len(batch) >= batch_capacity:
10            send_queue.put(batch)
11            batch = []

Agent.send_loop

POSTs batches; advances positions only after a 2xx ack

1def send_loop():
2    while running:
3        batch = retry_slot or send_queue.get()
4        resp = http.post(backend_url, batch)
5        if 200 <= resp.status < 300:
6            self.offset += sum(len(l) for l in batch)
7            persist(positions_yaml, self.offset)
8            retry_slot = None
9        else:  # network drop, 5xx, or no ack
10            retry_slot = batch  # resend after backoff
11            sleep(backoff_with_jitter())

Agent.on_restart

the only durable state is positions.yaml — RAM is gone

1def on_restart():
2    # batch and retry_slot lived in RAM — both gone
3    self.offset = read(positions_yaml) or 0
4    # Promtail gotcha: if positions advanced when the line
5    # was READ rather than when the backend ACKED, every
6    # in-flight line between offset and EOF is silently lost.
7    spawn(tail_loop)   # re-opens app.log, seeks to offset
8    spawn(send_loop)   # starts with empty queue + slot

Where this sits in Build a distributed logging stack (ELK / Loki)

Scene 02 of 12. A shipping agent tails each log file from a saved offset, batches lines, and POSTs them at-least-once — duplicates on retry are normal, not a bug.

Up next. The agent ships at-least-once over a pipe — but ships them WHERE? The backend at the far end can stall, and when it does the agent has exactly three options for what to do with the lines piling up — only one of which keeps the application alive.

All 12 scenes in Build a distributed logging stack (ELK / Loki) · Every curriculum

Built with Arqly

Every scene in Build a distributed logging stack (ELK / Loki) builds on the one before it.All 12 Build a distributed logging stack (ELK / Loki) scenes