#30Build a gRPC-style RPC framework
Every microservice talks over RPC, and the framework you ship determines half the system's failure modes. Build an RPC framework with codec, streams, deadlines, cancellation, retries, interceptors, and load-aware client-side balancing — and feel why gRPC ate the polyglot RPC market and why Thrift and JSON-over-HTTP linger.

You have written a Flask, Express, or Spring service. You have called another service over HTTP with a client library, and you have used a TCP socket at least once. That is your starting point. By the end of this curriculum you will have built — conceptually, layer by layer — a gRPC-style RPC framework, and you will be able to design an RPC interface for a concrete workload: pick a codec, pick a transport, set a deadline and a retry policy, choose how the client balances across backends, choose an mTLS posture, and predict what breaks under packet loss or a brownout — and defend every choice with a named scene.

We start from the wire, not from the framework. Scene 1 is a single uncomfortable fact: a remote call is dressed up to look like greet("Ada"), but unlike a local call it can fail after the server already did the work — so the caller can be left not knowing whether it happened. Every later feature in the stack exists to manage one of the four ways that illusion leaks: latency, no shared memory, partial failure, and unordered concurrency. We name that frame once and call back to it the whole way up.

This is a curriculum about a specific density-of-vocabulary discipline: across 14 scenes there are exactly 26 named technical terms, and each scene introduces at most two. The diagram carries the load; the vocabulary follows. One running example — the call greet("Ada") — is threaded through every layer: you watch it become protobuf bytes, ride inside a DATA frame on one of many streams, travel under a shrinking deadline header, and finally prove its own identity over mTLS.

Resist the urge to "describe gRPC." Build each layer because the previous layer forced it, feel the failure mode each one prevents, and let the design canvas at the end push back on your choices.

Reading: gRPC documentation — Core concepts (the four call types, metadata, channels): grpc.io/docs/what-is-grpc/core-concepts · gRPC blog — Deadlines ('always set a deadline'; the resource-exhaustion failure mode): grpc.io/blog/deadlines · gRPC guides — Retry policy (maxAttempts, retryable codes, backoff, throttling): grpc.io/docs/guides/retry · gRPC blog — Load balancing in gRPC (proxy vs client-side vs look-aside): grpc.io/blog/grpc-load-balancing · Kubernetes blog — gRPC load balancing on Kubernetes without tears (the L4 connection-pinning gotcha): kubernetes.io/blog/2018/11/07 · Birrell & Nelson — Implementing Remote Procedure Calls (TOCS 1984): the origin — stubs, marshalling, binding · Waldo, Wyant, Wollrath, Kendall — A Note on Distributed Computing (Sun TR-94-29, 1994): why RPC can't be transparent · Protocol Buffers — Encoding (varint, tag = field<<3|wire_type, skipping unknown fields): protobuf.dev/programming-guides/encoding · Protocol Buffers — Proto3 language guide (field numbers, reserved, schema-evolution rules): protobuf.dev/programming-guides/proto3 · Apache Thrift — Architecture / whitepaper (Slee, Agarwal, Kwiatkowski, Facebook 2007): the pluggable transport/protocol contrast · RFC 9113 — HTTP/2 (frames, streams, flow control, WINDOW_UPDATE, RST_STREAM): httpwg.org/specs/rfc9113 · RFC 9000 — QUIC & RFC 9114 — HTTP/3 (per-stream loss recovery; the TCP head-of-line fix) · Marc Brooker / AWS Builders' Library — Timeouts, retries, and backoff with jitter (retry storms, metastable failure) · Robin Marx — Head-of-line blocking in QUIC and HTTP/3 (app-layer vs transport-layer HOL): github.com/rmarx/holblocking-blogpost · Cindy Sridharan — Distributed Tracing in Practice (context propagation across hops)
the remote-call illusion: it looks like a local function but can fail after the work is done (partial failure)
framing on a TCP byte stream: recv() is not a message; length-prefix to find the boundaries
IDL + codec: protobuf field numbers (not names) as the schema-evolution superpower; never reuse a number
HTTP/2 as transport: one RPC = one stream; many streams multiplexed over one connection
unary, server-stream, client-stream, bidi — four shapes from one stream of DATA frames
deadlines, not timeouts: an absolute clock that PROPAGATES across an N-hop chain (the single most important feature)
cancellation as a first-class event (RST_STREAM); the Context object carries deadline + cancel together
retries: idempotency gates whether, a token-bucket budget + jittered backoff gate how much (retry storms / metastable failure)
interceptors / middleware: auth, metrics, tracing, retry composed as one onion around every call
client-side load balancing + discovery: why an L4 LB pins every stream to one backend (no central LB)
flow control / backpressure via the HTTP/2 window; a slow reader slows the writer
HTTP/2 fixed app-layer head-of-line blocking; TCP-layer HOL remains; QUIC/HTTP3 fixes it
TLS vs mTLS: channel encryption vs verified workload identity, propagated across the call chain