Fifty services, fifty broken retry policies

When every service re-implements retry, timeout, breaker, mTLS, and tracing in its own language, the network becomes non-uniform and a single wobbly dependency cascades into a fleet-wide outage.

Scene 01

Fifty services, fifty broken retry policies

Watch
Try it
Predict
Capture

What to watch for

Look at the badges. Twelve services, twelve different retry libraries — okhttp, axios, requests with unbounded retries, fetch with none. S7 in the middle just got slow (its box is red, tagged SLOW). Its three callers — S2, S5, S9 — feed straight into it. Nothing has cascaded yet.

Implementation

Highlighted lines are the ones running in the diagram right now.

Team checkout (Java) — retry in the app

OkHttp, exponential backoff, three tries — somebody's flavor

1Response call_profile(Request req):
2    for attempt in 1..3:
3        try:
4            return okhttp.newCall(req).execute()
5        except IOException:
6            sleep((2 ** attempt) * 100ms)
7    raise UpstreamFailed()
8# no shared budget · no breaker · no deadline

Team search (Python) — retry in the app

requests, bare except, infinite retries — different shop

1def call_profile(req):
2    while True:
3        try:
4            return requests.get(req.url, timeout=None)
5        except Exception:
6            continue  # try again forever
7# no backoff · no cap · no jitter · no timeout

Team recs (Node) — retry in the app

axios, fixed retry=5, no timeout — common footgun

1async function callProfile(req) {
2  return axios.request({
3    url: req.url,
4    // timeout: undefined  // forgot to set one
5    'axios-retry': { retries: 5 },
6  })
7}
8// fires 5 extra calls at a peer that's already slow

After the mesh — what the app keeps

the app becomes trivial · policy lives somewhere else

1async function callProfile(req) {
2  // localhost · the same in every language
3  return fetch('http://localhost/profile' + req.path)
4}
5# retry / timeout / breaker / mtls / tracing —
6# owned by the thing sitting next to the app

Where this sits in Build a Service Mesh (Envoy / Istio style)

Scene 01 of 13. Every team picks its own retry, timeout, breaker, and mTLS library. One slow dependency turns into a fleet-wide outage.

Up next. If the problem is that fifty app teams cannot agree on how the network should behave, the fix is to take the network out of the app's hands and put it in something all fifty share.

All 13 scenes in Build a Service Mesh (Envoy / Istio style) · Every curriculum

Built with Arqly

Every scene in Build a Service Mesh (Envoy / Istio style) builds on the one before it.All 13 Build a Service Mesh (Envoy / Istio style) scenes