Deadlines, not timeouts

Build a gRPC-style RPC framework (14 scenes)

Scene 06 · Deadlines, not timeouts

A timeout is relative and resets each hop; a deadline is absolute and propagates the time remaining — so a downstream service never works for a caller that already gave up.

Previously

We saw one server keep computing 'Hello Ada' long after its wire died. Now stretch that across an A → B → C chain: without a shared clock, the deepest service keeps burning CPU for a request its caller abandoned a moment ago.

Scene 06

Deadlines, not timeouts

Watch

Diagram

A 3-hop call chain A → B → C, each arrow carrying greet("Ada"). The bar across the top is the shared budget that rides WITH the call and shrinks hop by hop; the chip above each arrow is the grpc-timeout header — the remaining absolute time handed to the next hop. A TIMEOUT is a relative duration ('fail after 1s') that resets fresh at every hop; a DEADLINE is one absolute moment in time that PROPAGATES, so B passes C only what's left. When work outlasts the shared clock the chain stamps DEADLINE_EXCEEDED; without propagation, C burns CPU on 'work for a ghost' — zombie work.

Sources

deadline = one absolute clock for the whole call →

grpc-timeout: remaining time handed to C ↓

A network call usually rides under a patience limit: 'give up if no answer comes back in time.' The naive way to express that is a TIMEOUT — a relative duration like 'fail after 1 second.' The catch is that a timeout RESETS at every hop: A waits 1s on B, but when B calls C, B starts C's clock fresh at another full second — so the chain as a whole can wait far longer than anyone intended. The fix is a DEADLINE: instead of a duration, you compute one absolute point in time ('give up at 12:00:01.000') and PROPAGATE it. Each hop subtracts the time already spent and hands the next hop only the remaining time. Watch greet("Ada") travel A → B → C under a 1000ms deadline: the budget bar up top shrinks as time is spent, and the grpc-timeout chip on each arrow shows the smaller 'time left' — one absolute clock, re-expressed per hop.

Implementation

Client.call

each hop re-expresses the same absolute deadline as time left

1def call(stub, req, ctx):
2    # remaining time on the ONE shared clock, not a fresh duration
3    remaining = ctx.deadline - now()
4    headers = {
5        'grpc-timeout': encode(remaining),
6    }
7    return stub.invoke(req, headers)

Server.handle

the receiver rebuilds an absolute deadline from the header

1def handle(req, headers):
2    if 'grpc-timeout' in headers:
3        # honor the caller's clock — bind to the same moment
4        ctx.deadline = now() + decode(headers['grpc-timeout'])
5    else:
6        # no propagation: start a fresh full timeout, alone
7        ctx.deadline = now() + DEFAULT_TIMEOUT
8    result = doWork(req, ctx)
9    return result

Server.doWork

work races the deadline; losing it is DEADLINE_EXCEEDED or zombie work

1def doWork(req, ctx):
2    while not done:
3        if now() >= ctx.deadline:
4            raise DEADLINE_EXCEEDED  # stop, caller is gone
5        step()  # one slice of the cWorkMs of work
6    return Reply(greeting=greet(req.name))

Not sure what to ask? Tap a question — the staff engineer answers in the chat panel.

PreviousFour call shapes from one stream NextCancellation: an event, not a clock