Origin shield — collapsing the herd — Build a CDN

Scene 09 · Origin shield — collapsing the herd

Without a shield, every POP independently misses on TTL expiry and stampedes origin in parallel; with a shield, origin sees one request instead of N.

Previously

Even with a perfect cache key, a popular object expiring across hundreds of POPs at once stampedes origin — unless we put one POP between them.

Scene 09

Origin shield — collapsing the herd

Watch

Diagram

A popular URL's TTL ticks down at the top; the row of lower-tier POPs in the middle each hold their own copy. When TTL hits zero every POP misses at the same instant — with shield OFF every miss arrow lands directly on origin (the gauge spikes to N), with shield ON the misses converge at one upper-tier POP that forwards a single request to origin (gauge reads 1) at the cost of one cross-region hop. **thundering herd** — many edge caches missing the same key at the same instant (typically TTL boundary) and all fanning out to origin simultaneously. **origin shield** — a designated upper-tier POP that absorbs lower-tier misses so origin sees one request instead of N.

Sources

Shield is OFF. The popular URL is cached at every POP and a TTL countdown is ticking on every cell. When the countdown hits zero, every POP misses at the same instant — watch what happens to the origin gauge.

Implementation

POP.on_miss (shield OFF)

every POP fans out to origin on its own miss

1def on_miss(req):
2    # cell expired or never populated here
3    upstream = origin
4    resp = upstream.fetch(req.url)
5    cache.put(
6        req.cache_key, resp,
7        ttl = resp.cache_control.max_age,
8    )
9    return resp
10    # at TTL boundary every POP runs this in parallel
11    # → origin sees N simultaneous fetches for one byte

POP.on_miss (shield ON)

lower tier asks the designated upper-tier POP first

1def on_miss(req):
2    # ask the shield POP, not origin
3    upstream = shield_pop_for(req.cache_key)
4    resp = upstream.fetch(req.url)  # +cross-region hop
5    cache.put(
6        req.cache_key, resp,
7        ttl = resp.cache_control.max_age,
8    )
9    return resp

ShieldPOP.fetch

request coalescing — N concurrent misses, 1 origin fetch

1inflight = {}  # cache_key → Future
2 
3def fetch(url):
4    key = cache_key(url)
5    if key in cache and not cache[key].stale:
6        return cache[key]
7    if key in inflight:
8        return inflight[key].await()  # piggyback
9    inflight[key] = spawn(origin.fetch(url))
10    resp = inflight[key].await()
11    cache.put(key, resp)
12    del inflight[key]
13    return resp

PreviousCache key and Vary — when 'same URL' isn't NextBypass — when caching is wrong, the CDN still earns its keep