POPs and anycast — one IP, many doors

Scene 03 · POPs and anycast — one IP, many doors

Every POP advertises the same IP; the network's BGP routing fabric, not a CDN dispatcher, picks which edge each user reaches.

Previously

Edges only help if users actually land on a nearby one — so how does a user in Sydney find Sydney's edge with no load balancer in the middle?

Scene 03

POPs and anycast — one IP, many doors

Watch

Diagram

World map with six POP markers (LA, Frankfurt, Sydney, São Paulo, Singapore, Mumbai); each POP advertises the SAME anycast IP (104.16.0.1) shown as a literal next to the marker. User bubbles fire requests; a routing arrow snakes through one or two ISP/transit hops and lands on a POP. The chosen POP is ringed and labeled with the shortest BGP path — the network picked it, not a configured load balancer.

Sources

Every POP advertises the SAME IP (104.16.0.1). The user's ISP — not the CDN — picks the one with the shortest BGP path. The chosen POP is ringed.

A user in Sydney fires a request to 104.16.0.1. Notice every POP on the map advertises the same IP. The user's ISP picks the shortest BGP path — that's the Sydney POP, ~15 ms away.

Implementation

AuthoritativeDNS.resolve

every user gets the SAME IP — there is no per-region answer

1def resolve(name):
2    # cdn.example.com -> 104.16.0.1 for EVERYONE
3    # no geo-IP lookup, no per-region answer
4    return ANYCAST_VIP  # 104.16.0.1

ISPRouter.routePacket

BGP picks the shortest AS-path to the anycast prefix

1def routePacket(pkt):
2    # bgp_table[prefix] -> list of (nextHop, asPath)
3    routes = bgp_table[pkt.dstPrefix]  # 104.16.0.0/24
4    best = min(routes, key=lambda r: len(r.asPath))
5    forward(pkt, nextHop=best.nextHop)
6    # The CDN never saw this decision. Each ISP made
7    # it independently from its own BGP table.

POP.bgpLifecycle

a POP joins or leaves the anycast group by announcing or withdrawing the prefix

1def comeOnline():
2    # tell upstream peers 'I have a path to 104.16.0.0/24'
3    bgp.announce(ANYCAST_PREFIX, asPath=[self.asn])
4 
5def goOffline():
6    # withdraw the route — peers stop seeing us
7    bgp.withdraw(ANYCAST_PREFIX)
8    # ISPs recompute; users land on next-shortest POP.
9    # No client change. No DNS change. No CDN dispatcher.

PreviousAn edge near the user — hit and miss NextTTL — origin tells the edge how long to trust the copy