Concurrent Hotel Viewers — a worked solution

What each component is for

Client (page)

Renders the hotel detail page; on load fires POST /presence/heartbeat every 15s; subscribes via WebSocket to count.{hotel:H} for live updates and falls back to a 10s GET /presence/count poll if the WS handshake fails. Tags every request with a fingerprinted session cookie set on first paint.

Why it exists. It is the source of every write and read in this design — the heartbeat *is* the presence signal. We chose a client-pushed heartbeat over server-tracked TCP-state (the BookingChat-style 'count subscriber-set on the gateway') because TCP-state breaks the moment a CDN sits in the middle: the gateway sees PoP IPs, not user identities, and you've inflated your count to 200.

When it fails. If the heartbeat fails, the user's own count drifts down by one within 30s; if the count widget itself fails, render —. The page MUST NOT be on the critical-rendering path of the count widget — synthetic checks watch for blank-page time on hotel detail pages and block any deploy that puts the count XHR on a await before first paint.

CDNFastly (VCL) + origin shield

Serves GET /presence/count/:hotel from edge cache with 5s TTL and stale-while-revalidate up to 30s. Forwards POST /presence/heartbeat uncached but with WAF + per-IP/ASN rate-limit. Terminates TLS, ships logs to the abuse pipeline. Origin shield centralises misses to one PoP per region so the origin sees one miss per region per TTL window, not one per edge POP.

Why it exists. The cardinal trick of this whole design: with T_edge=5s, origin reads per hotel are bounded at 1/T_edge = 0.2 reads/s regardless of viewer count. Without the CDN, 5M concurrent viewers polling every 10s = 500K reads/s slamming the count service. We considered putting count behind an internal cache only (skip the CDN), but that means every viewer pays a full origin RTT per poll — UX research shows the count widget perceptibly slows the page above 60ms.

When it fails. PoP outage (Fastly 2021-06-08, 49 min global): origin sees 20–50× normal QPS within seconds. Detection: cdn_5xx_rate{pop=$x}>5% for 2m. Mitigation tiered — (1) origin must hold ≥3× steady-state for 60 min, (2) the count widget JS catches and shows —, (3) regional DNS failover to alternate edge provider. The widget MUST NOT block the page if the CDN is down.

Function (FaaS) · Edge AggregatorCloudflare Workers + Durable Objects (per-PoP)

Two responsibilities, one runtime. WRITE PATH: receives POST /presence/heartbeat at the PoP, verifies the session-fingerprint cookie signature (rotating Ed25519 verify key pushed to every PoP — verification needs no signing secret) and applies the PoP-local 10 heartbeats/sec per-fingerprint token bucket (fingerprints are PoP-sticky, so local enforcement is global enforcement), computes x-fingerprint-h64, holds the (hotel_id, fingerprint) tuple in an in-memory PoP-local Bloom filter, and once per flush window per (hotel_id, PoP) flushes a single PFADD {hotel:H}:shard:R hash1,hash2,... batch upstream, tagged with per-ASN entry counts for the gateway's global buckets. Collapses each hot (hotel, PoP) pair's writes to ≤1 batched call/s. READ PATH: on cache miss, applies XFetch probabilistic-early-expiry and singleflight coalescing — all simultaneous misses for the same hotel within one PoP collapse to one origin fetch.

Why it exists. Without edge fan-in, viral-hotel write QPS bottoms out the hot Redis slot at V_viral / H = 6.6K writes/s for V=100K — and scales linearly to catastrophe at V=1M. The edge layer absorbs that fan-in: 280 PoPs × 1 batched write/s = 280 batched writes/s on the hot slot regardless of V_viral. We considered (a) sharding the hot key into N parallel HLLs at origin (still does N writes per heartbeat) and (b) client-side sampling (loses precision below 1K viewers). Edge fan-in is the only option that avoids per-heartbeat origin writes AND keeps a flat origin write rate — at the cost of the PoP Bloom's bounded ~1% undercount, still well inside the ±5% SLO (unlike client sampling, which degrades badly below 1K viewers).

When it fails. Cold PoP (newly-spun region, post-deploy bounce): no Bloom state, no DO warmth — every heartbeat goes through unbatched. Origin sees a transient ~24× write spike on hot slots (280 batched writes/s → up to V_viral / H ≈ 6.7K/s) for ~30s. Mitigation: the LB capacity-plans for 3× steady-state and the heartbeat path circuit-breaks at 50K writes/s per upstream LB pool. Health signal: edge_fn_dispatch_p50{pop=$x} plus do_storage_hit_ratio{pop=$x}. We accept that for a region's first 30s, count freshness lags by min(H, batch_window) ≈ 1s — invisible at the 5s edge cache TTL.

Load BalancerEnvoy ring-hash tier (Global Accelerator anycast front, NLB internal)

Anycast-fronted L7 ingress; routes POST /presence/heartbeat/:hotel by consistent-hash(hotel_id) to the heartbeat-service pool, and GET /presence/count/:hotel by least-conn to the count-service pool. Health-checks with /healthz?verify=cluster once per second; ejects unhealthy targets within 10s. TLS 1.3, mTLS to all internal pools. Caps origin connections per upstream pool at 50K writes/s (slot-ceiling head-room) and sheds with 503 + Retry-After jitter when exceeded.

Why it exists. Read traffic is symmetric — least-conn is fine. Write traffic isn't: routing heartbeats by hotel_id to the same heartbeat-service replica means each replica's local hot-key Bloom and per-(hotel, shard) write batcher actually function. Without affinity, the per-replica batchers see fragments of the traffic and you lose the per-replica coalescing on top of edge coalescing. We considered putting the affinity at the gateway (slower, more L7 cost), but the LB is already terminating connections — cheapest place.

When it fails. Single-AZ LB outage: anycast re-converges in ~10s; in-flight requests 5xx. Detection: lb_target_5xx_rate>1% for 30s. Heartbeats are retry-tolerant (next one in 15s); count reads ride SWR=30s. RTO 10s automatic, no data loss (LB is stateless). Cold-PoP affinity collapse (the load-bearing failure mode): when edge-fn is cold in a region, the ~24× spike documented on edge-fn.failureMode (280 batched writes/s → up to V_viral / H ≈ 6.7K/s) collides with consistent-hash(hotel_id) — ALL writes for a single hotel land on ONE hb-svc replica that must briefly absorb that ~6.7K writes/s (vs its ~1.3K/s nominal share). Mitigation is dual: (1) per-target circuit-breaker on e5 lb→gw and on the hb-svc upstream pool ejects the saturated replica within 5s — heartbeats for that hotel decay for ~10s during the gap (visible count drift, NOT a page break); (2) the LB falls back from consistent-hash to least-conn for the affected hotel via the x-fallback: 1 header set by edge-fn when its local Bloom is empty, distributing load at the cost of breaking per-replica batching for that hotel. Pages on-call when lb_per_target_breaker_open_rate>0.5% for 2m.

API GatewayEnvoy + custom WASM filter (fingerprint validation)

On the coalesced write path, per-fingerprint HMAC validation and its 10-heartbeats/sec token bucket already ran at edge-fn (PoP-local, before coalescing), so the gateway consumes the per-ASN entry counts edge-fn tags on each batch and enforces the GLOBAL rate-limits that need cross-PoP coordination — 1000 heartbeats/sec per ASN and per-IP-/24 (a botnet spread across 280 PoPs is invisible to any single PoP's local bucket). Strips PII headers and attaches x-bucket-30s at the gateway clock so client clock skew can't shift the HLL bucket. On the cold-PoP fallback path (edge-fn bypassed, unbatched heartbeats) it does the full per-request check itself: validates the session-fingerprint cookie (HMAC of cookie || IP-prefix-/24 || UA-fingerprint, signed with rotating Ed25519) and computes x-fingerprint-h64 (BLAKE3 of cookie, 64-bit) — the hash, not the cookie, is what gets PFADD'd.

Why it exists. Heartbeat is unauthenticated by design (anonymous browsing of hotel pages must work), so we can't hide behind bearer tokens. The gateway is the only layer that can cheaply do per-(IP, ASN) rate-limit with global coordination — putting it at the edge means each PoP rate-limits independently and a botnet across 280 PoPs gets 280× headroom. We considered moving fingerprint validation into the heartbeat service (saves a hop) but then every replica needs the rotating signing key and the blast radius of a key leak grows from 12 → 64 nodes.

When it fails. Rate-limit Redis (small, separate cluster) failure: the gateway fail-opens — accepts the heartbeat without rate-limit. Detection: gw_rl_redis_timeout_rate>0.5% for 1m. Blast radius is bounded — even an unlimited heartbeat rate is capped per-PoP by the LB connection ceiling (50K writes/s), so a botnet can spike origin briefly but can't melt it. If the *signing key* leaks, we rotate immediately and accept that all sessions need to re-handshake — count drops by ~30% for ~10s as old fingerprints get rejected.

Service · HeartbeatRust + Tokio (async runtime, hot path)

Receives POST /presence/heartbeat already pre-coalesced by the edge layer; looks up hotel shard config in meta-db (cached locally with 60s TTL); computes {hotel:H}:shard:rand(0..N_H-1):bucket:{floor(now/30)} where N_H ∈ {1, 16, 64} per hotel's hot-key tier; PFADD's the fingerprint hash. Emits one event per heartbeat to presence.heartbeats (Kafka) for abuse + analytics. Total path is ≤ 30ms p99; lock-free in the hot path; no synchronous fan-out.

Why it exists. We split heartbeat from count because their failure modes are different: heartbeat loss is invisible (next one in 15s); count read failure is visible (page shows —). Splitting lets us scale them independently — the raw heartbeat aggregate is 333K writes/s, but edge fan-in reduces what hb-svc actually receives to ~83K req/s (5M req/min), which 64 replicas handle at 1.3K req/s/replica; count at 50K reads/s post-edge needs 8. Sharing a service would force the same replica count and saturate the count path during write spikes.

When it fails. Per-replica deadlock or memory leak: with consistent-hash routing, a single bad replica drops 1/64 of heartbeats — a 1.6% precision hit, invisible at HLL's 0.81% baseline error. The replica autoscaler (HPA on hb_writes_inflight) provisions a replacement in ~45s. Worst case is *all* 64 replicas crash simultaneously (deploy bug): heartbeat path dark for ~3min until rollback, counts decay to zero in 30s, page shows —. The page does not break.

Service · CountRust + Tokio + tower-singleflight

On GET /presence/count/:hotel cache-miss from edge, computes the live count via PFCOUNT {hotel:H}:shard:0..N_H-1:bucket:{cur} UNION {hotel:H}:shard:0..N_H-1:bucket:{prev} — two-bucket sliding window. Applies XFetch (Vattani 2015) probabilistic early expiry to refresh hot keys before the edge TTL hits zero. Singleflights all in-flight requests for the same hotel via tower-singleflight so origin sees one PFCOUNT per hotel per second. Publishes count deltas to count-pubsub for the WS push fleet.

Why it exists. PFCOUNT is O(N_shards) — for N=64 it's 64 register reads + 1 union, all within one Redis slot thanks to the hashtag pinning. Doing this on every request from the edge would hammer the slot. We considered (a) caching PFCOUNT result inside count-svc (race-with-write, stale by up to 5s), (b) precomputing on a write-time keyspace-notification listener (extra Redis dependency, complex). XFetch + singleflight inside count-svc is the cleanest: one Redis read per hotel per second, served to all simultaneous requesters from the in-flight future.

When it fails. Redis hot-shard contention spike (viral hotel): PFCOUNT p99 climbs from 8ms → 80ms; the breaker on e8 opens fast and we serve last-known-good from the 1s in-process cache. Detection: count_svc_redis_p99>30ms for 2m (auto-mitigates) → breaker_open_rate{target=redis-hll}>5% for 5m (pages on-call). The known hand-wave: tower-singleflight is per-REPLICA, not cluster-wide — across 8 replicas during rollout or least-conn fan-out, a hot hotel can land 8 concurrent PFCOUNTs on the slot. We accept that 8× burst as part of the redis-hll mitigation tree (the slot's headroom is sized for it: 50K writes/s ceiling vs 280 batched origin writes/s steady-state) rather than push singleflight into Redis via Lua. Worst case all PFCOUNT calls timeout: widget renders stale count with ?approximate=true&stale=true; page never blanks.

Service · WS/SSE FanoutGo + nhooyr/websocket, sticky-hash routing

Holds long-lived WebSocket connections from clients (SSE fallback for restrictive networks). Subscribes to count-pubsub channels for hotels with active subscribers. Coalesces broadcasts per Pusher's published rules: hotels with ≤100 subscribers broadcast every count change; hotels >100 broadcast at most once per 5s with bundled deltas (matches Pusher's published 5s lock + 30s bundle window). Routes incoming subscriptions by consistent-hash(hotel_id) so all subscribers for the same hotel land on the same replica — Discord's 'guild → gateway shard' pattern translated to hotels.

Why it exists. Pure polling at 5M concurrent pages × 1 read / 10s = 500K reads/s; even after 90% edge absorption, that's 50K reads/s flowing through count-svc constantly, including for hotels nobody is watching. WS fanout flips the model: a hotel with active viewers gets one push per ~5s regardless of subscriber count. We considered SSE-only (one-way, no reconnect handshake); we kept WS because we need the client→server channel for keep-alive frames that prove connection liveness — these carry no presence; presence is written solely by the separate POST /presence/heartbeat on its own path, keeping the read (push) and write (heartbeat) planes independent.

When it fails. Reconnect storm after rolling deploy (Slack 2018-06-27 pattern): 5M clients reconnect within 30s after fleet bounce. Detection: ws_fleet_handshake_rate > 5× baseline. Mitigation: server sends Reconnect-After: $jittered_seconds in the close frame (decorrelated jitter, AWS Builders' Library); rolling deploy paces ≤ 5% of fleet at a time; LB sheds new connections at 1.2× steady-state. Cert expiry on the WS edge: hard outage, all clients fall back to 10s polling. We have cert-expiry alerts at T-30/14/7/1 days (post-Spotify 2022).

Cache · Redis HLLRedis 7.4 Cluster, m7g.4xlarge × 16

Stores per-hotel HyperLogLog state at precision 14 (16384 registers, 12 KB/key, 0.81% standard error per Heule et al. 2013). Two buckets per hotel for the sliding window (bucket:{floor(t/30)}, bucket:{floor(t/30)-1}). For viral hotels, splits each bucket across N ∈ {16, 64} parallel HLLs co-located in one slot via the {hotel:H} hashtag — this lets PFCOUNT/PFMERGE remain atomic and intra-slot, while spreading PFADD writes across N register-arrays' CPU. TTL = 60s on every key.

Why it exists. HLL gives us the cardinality of a viral hotel for 12 KB regardless of viewer count — 100K viewers and 100M viewers cost the same memory. The 0.81% standard error is well below the product's ±5% SLO. We rejected ZSET (exact, but per-element 80 B → 4 MB at V=50K with O(log N) ZADD on the hot slot) and Memcached CAS counters (no probabilistic primitive, would need app-level HLL outside the store). Redis HLL is in-engine: PFADD is one O(1) register update.

When it fails. Hot-shard CPU pegged on viral hotel (Booking.com-style spike): one slot at 100% while cluster sits at 5%. Detection: redis_cpu_seconds{slot=$hot}>0.9 while cluster_cpu<0.10 for 30s — drives hotkey-controller auto-promotion. Two-stage mitigation: (1) controller bumps N_shards via meta-db (count-svc picks up first, hb-svc 90s later — see keyChoices); (2) edge fan-in cadence drops 1s → 250ms. CLUSTER REBALANCE on the hot slot: PFADD/PFCOUNT return ASK redirects during CLUSTER SETSLOT MIGRATING/IMPORTING; clients follow and counts continue, but the 'atomic intra-slot PFMERGE' guarantee briefly breaks. We gate slot-migration on hot-tier hotels via a slot-pin policy in hotkey-controller and a runbook that disables auto-rebalance for slots holding any V≥10K hotel — migration becomes operator-driven planned-maintenance. Worst case slot loss: leader-failover RTO=15-30s during which PFADDs 503; widget shows last-cached count for the gap, then resumes. RPO=5s of heartbeats lost — invisible because next heartbeat is in 15s and HLL is idempotent on retry. Pages on-call when failover RTO > 60s OR when hot-shard CPU>0.9 sustained 3m without controller mitigation; otherwise auto-mitigates.

Stream · Count Pub/SubRedis 7.0 Sharded Pub/Sub (SSUBSCRIBE)

Fast-path broadcast channel for count deltas. Count-svc SPUBLISH count.{hotel:H} <count> on every recompute; the WS fanout fleet SSUBSCRIBEs only for hotels with active local subscribers. Sharded Pub/Sub (Redis 7.0+) routes by hashtag so subscribers connect to the slot owner directly — no proxy hop. Zero retention; subscribers that disconnect don't replay (this is fine: the next count read or next push will catch them up).

Why it exists. We need a fan-out channel for count → ws-fanout that does not couple their lifecycles. Without pub/sub, count-svc would need a registry of active subscribers per hotel (load-bearing state), or ws-fanout would poll count-svc per hotel per cycle (rebuilds the read fanout problem we just solved). Redis Sharded Pub/Sub piggy-backs on the cluster we already operate, with one hop. We rejected Kafka here — Kafka topics with retention=0 are wasteful, and Kafka's per-topic overhead at 10M hotel-channels would be 10M consumer-group offsets, untenable.

When it fails. Subscriber-side lag (ws-fanout replica GC pause): SUBSCRIBE buffers fill, Redis evicts the slow subscriber per client-output-buffer-limit pubsub 32mb 8mb 60. Detection: pubsub_dropped_clients>0. Mitigation: ws-fanout replicas auto-resubscribe and the next count tick pushes again — at most 5s of staleness. Pub/Sub server outage: ws-fanout falls back to polling count-svc directly at 5s/hotel. Capacity headroom on count-svc is 5×, so this is survivable for the duration of the failover.

Stream · Heartbeat EventsApache Kafka 3.7, MSK m7g.2xlarge × 9

Persistent event log of every heartbeat ({hotel_id, fingerprint_h64, asn, geo, ts}) — *not* the HLL writes, just the metadata for off-line scoring. Kafka topic presence.heartbeats, 64 partitions keyed by hotel_id, RF=3, acks=all on producer, 72h retention. Feeds the abuse worker (real-time scoring), the analytics warehouse (Snowflake nightly batch), and a synthetic-traffic detector. Producer is fire-and-forget — hb-svc returns 204 to the client before the Kafka ack.

Why it exists. We need durable, replayable, ordered metadata for abuse detection that the volatile HLL Redis intentionally throws away. Putting abuse scoring in the hot path (hb-svc → meta-db sync write) would double heartbeat p99 and couple the count's availability to the metadata DB's. The stream gives us 24h of replay if a scoring rule changes, without re-processing live traffic. We rejected (a) writing scoring features to meta-db on each heartbeat (sync coupling) and (b) sampling — sampling misses the rare attacker patterns we need to catch.

When it fails. Broker outage (single-AZ): producer retries to in-sync replicas, no client-visible impact (acks=all + min.isr=2 survives 1 broker loss). Consumer-side lag (abuse worker falls behind): degrades scoring freshness, doesn't break heartbeats. Detection: kafka_consumer_lag{topic=presence.heartbeats}>1M for 5m. Worst case complete topic loss: scoring degrades but heartbeats and counts work normally. The cardinal contract: this stream's failure must NOT be visible on the page.

Worker · Abuse ScannerGo + sliding-window counters in BadgerDB

Consumes presence.heartbeats partitioned by hotel_id; maintains per-(fingerprint, ASN, hotel) sliding-window counters in embedded BadgerDB; scores patterns (heartbeat-rate spikes from unseen ASNs, fingerprint reuse across geo-impossible IPs, UA fingerprints with rotating cookies) using a hand-tuned rule set + a downstream gradient-boosted model that gets updated weekly. On a positive verdict, writes (fingerprint, expires_at, reason) to meta-db's abuse_denylist table — hb-svc reads this with its 60s cache.

Why it exists. Bot armies inflate counts (or deflate competitor hotels) — we need a feedback loop that makes the heartbeat path hostile to attackers without making it expensive for honest users. Putting scoring in the hot path costs every legitimate user a model inference (~1ms) for the rare attack signal. The async stream lets us spend ~250ms per heartbeat on scoring without anybody noticing. Hot-path denylist lookup (the cheap part) stays in the gateway via meta-db cache.

When it fails. Worker crash loop on poison message (e.g. malformed UA from a buggy SDK): one partition stalls, consumer-group lag grows. Detection: kafka_consumer_lag{partition=$x}>1M for 10m. Mitigation: DLQ pattern — after 3 retries, route to presence.heartbeats.dlq and continue. Worst case all 12 replicas stall: scoring goes silent for hours, attackers get a free window. The page does not break — denylist lookups still hit the last known state in meta-db. We page on-call when consumer lag exceeds 30 min.

SQL DB · MetadataPostgres 16 HA (Patroni + 2 sync followers)

Authoritative source for low-write, high-read metadata: hotel registry (hotel_id, name, region), per-hotel hot-key config (n_shards_read / n_shards_write ∈ {1, 16, 64}, last-promoted ts), abuse denylist (fingerprint_h64, expires_at, reason). Read by hb-svc on every heartbeat (cached 60s), by gateway for fingerprint-revocation checks (cached 5s), by abuse worker for scoring rules. Write paths: a control-plane operator service auto-promotes hotels between hot-key tiers based on observed write QPS (metric → control loop → 2-phase UPDATE hotel_shards).

Why it exists. Some metadata genuinely needs durability — abuse denylist must survive Redis bounces, hot-key tier decisions need a record so hb-svc and count-svc agree on N_shards, hotel registry is shared with the broader booking platform. Postgres for the same reason every team picks it: SQL escape hatch for ad-hoc operator queries (SELECT hotel_id, count(*) FROM abuse_denylist WHERE created_at > now()-'1 hour'::interval), and we already operate it. We rejected DynamoDB (rigid query patterns) and putting this in Redis (no durability semantic for abuse denylist).

When it fails. Primary failover (Patroni-orchestrated): writes block for ~30s; hb-svc and gateway continue reading from local 60s caches — no user-visible impact. Detection: pg_replication_lag>5s or patroni_state!=running for 1m. Worst case primary + both sync followers down: control-plane writes block until manual promotion; hot-key tier promotions and abuse denylist updates pause; hb-svc and gateway run on stale config for hours, which is fine — counts stay accurate, attackers caught yesterday remain caught.

Cache · Rate-Limit RedisRedis 7.4 single primary + 1 follower (NOT clustered)

Backs Envoy's ratelimit-service. Holds the short-TTL GLOBAL token-bucket counters the gateway coordinates — keyed by (IPv4-/24) and (ASN) — two bucket queries per gateway check (once per batch on the coalesced path; the per-fingerprint bucket lives PoP-local at edge-fn, not here). Returns ALLOW/DENY in <2ms p99 over a connection-pooled TCP path; gateway fail-opens (ALLOW) on timeout >5ms.

Why it exists. Putting rate-limit state in the same Redis cluster as the HLL data means a hot-shard meltdown on a viral hotel's HLL slot also melts rate-limit reads for every other hotel — one viral incident knocks out global abuse defense. Splitting blast radius is the load-bearing reason this is a separate cluster, even though it costs operating two Redises. Single-instance + 1 follower (not clustered) because the working set (per-/24 and per-ASN buckets with 60s TTL, ~100MB) fits in one node and clustering would just add slot-routing cost.

When it fails. RL Redis failure: gateway fail-opens, accepts every heartbeat without rate-limit. Detection: gw_rl_redis_timeout_rate>0.5% for 1m. Blast radius bounded — even unlimited rate, the LB connection ceiling (50K writes/s per upstream pool) caps per-pool damage; HLL itself is idempotent on the same fingerprint. Pages on-call when gw_rl_redis_timeout_rate>5% for 5m (sustained fail-open is bot-army-friendly and we want a human to look). Single-AZ failure: Sentinel promotes the follower in 10s; brief <1s write-stall at the moment of cutover during which a few heartbeats fail-open.

Service · Hot-Key ControllerGo + leader-elected via Patroni advisory lock on meta-db

Continuously samples per-slot write QPS and CPU from obs (PromQL: sum by (slot) (rate(redis_pfadd_total[30s])) + redis_cpu_seconds); when a hotel's slot exceeds the (N_shards, V_viewers) threshold table — 1→16 at V≥1K OR pfadd_qps>1K, 16→64 at V≥10K OR pfadd_qps>10K — issues the 2-phase tier promotion: STEP 1 UPDATE hotel_shards SET n_shards_read=$new (count-svc snapshot picks up at next 60s rotation), wait 90s, STEP 2 UPDATE hotel_shards SET n_shards_write=$new. Demotes (64→16→1) symmetrically after V drops below tier-floor for >10 min. Also enforces the slot-pinning runbook for V≥10K hotels (refuses a manual CLUSTER REBALANCE request on those slots).

Why it exists. The auto-promotion N=1→16→64 is the load-bearing safety story for hot keys, and we need a service that *runs* it — not implicit logic on hb-svc (each hb-svc replica would race) and not a manual operator runbook (60s reaction time is the difference between healthy auto-promotion and a melted slot). Leader-elected so only one replica drives the control loop at a time; the 2nd replica is hot-standby with a 5s lock-acquisition timeout. The promotion threshold V=10K (not V=50K) gives the metric-loop budget headroom: slot saturates in 5s at V_viral, our scrape interval is 15s — promoting at V=10K leaves 4× margin while the slow loop catches up.

When it fails. Both replicas crash: promotion stops; hot keys saturate without intervention. Detection: hotkey_controller_leader_age_seconds > 60. Pages on-call within 60s — this is *the* control-plane that prevents melt-downs and a human MUST take over with manual UPDATE hotel_shards. Replication-lag > 30s on the meta-db read replica controller uses for its sample: controller refuses to issue promotion (stale data is worse than no promotion); pages immediately. Detection: hotkey_controller_data_age_seconds > 30.

Monitoring · ObservabilityPrometheus 2.x + Thanos + Grafana + Alertmanager + PagerDuty

Scrapes /metrics from every service/cache/stream/db node every 15s; long-term storage in Thanos with 30-day retention. Hosts the SLO dashboards (count-read p99, heartbeat-write success-rate, hot-shard CPU heatmap, ws-fanout connection count, abuse consumer lag). Drives Alertmanager → PagerDuty for the page-on-call thresholds named in each node's failureMode. Exposes a query API the hotkey-controller consumes for its 15s control loop.

Why it exists. Every failureMode in this canonical names a metric (cdn_5xx_rate, redis_cpu_seconds, count_svc_redis_p99, kafka_consumer_lag, gw_rl_redis_timeout_rate, pg_replication_lag, ws_fleet_handshake_rate, lb_per_target_breaker_open_rate). These metrics drive auto-mitigation (controller-driven shard promotion, breaker decisions, autoscale signals) — they are not optional decoration. Putting obs on the diagram makes the metric pipeline drawable; without it, the chaos scenarios that say 'Detection: $metric' have no graph target. Prometheus over Datadog because we already operate it; Thanos for cross-region federation when we add multi-region.

When it fails. Prometheus replica failure: 6-replica federation; Thanos sidecar fronts queries — single replica loss is invisible. Quorum loss (4/6 replicas down): metrics gap until quorum recovers; Alertmanager flushes pending alerts on recovery. Detection: prometheus_up < 4 for 1m. Pages immediately because we lose detection on every other failure. Worst case complete obs outage: the controller can't promote (refuses on stale data, see hotkey-controller); manual operator promotion via runbook; slot melts only if a viral hotel happens to spike during the obs gap, which is the genuine 3am page.

Service · Peer Region CountStub representing a peer-region count-svc (full mirror of this canonical)

Represents one peer region's count-svc for the cross-region count-merge story. Local count-svc fetches GET /presence/count-merge-bundle/:region from each peer region every 1s for hotels with cross-region viewers, PFMERGE's the snapshots, returns the union count. Stub-modeled here as one 'peer' node so the multi-region chaos scenario (region-partition-split-brain) has a graph target — in reality this is a full mirror of the entire canonical in another region.

Why it exists. The non-functional spec claims active-active per-region with ≤1s cross-region staleness. A single-region canonical can't defend that. Stubbing a peer-region node is the minimum graph representation needed to: (a) draw the cross-region merge edge that count-svc consumes, (b) let the chaos region-partition-split-brain exercise a real path. We considered mirroring the entire canonical for each region (infeasible — 17 nodes × N regions = unreadable diagram) vs downgrading to single-region (loses the active-active claim). The stub is the honest middle.

When it fails. Cross-region link partition: peer-region request times out at 1000ms; count-svc returns local-region-only count (visible drift in 'global' display proportional to peer-region's share of viewers). Detection: cross_region_merge_timeout_rate>10% for 1m. Pages on-call when sustained 5m — most cross-region partitions are <1m blips, and we accept regional-only display for the duration. Split-brain on heal: the redis-hll PFMERGE is associative and commutative — counts converge upward to UNION on next merge cycle, no manual reconciliation needed.