Live Comments / Score Updates — a worked solution

Clarifications worth surfacing on day 1 — every one of these flips a topology decision:

• Mega-room ceiling. Is the marquee event 1M concurrent (typical viral stream), 5M (Super Bowl / MrBeast / WC final), or 30M+ (cricket IPL final in India)? The Manifold relay-tree activation threshold and the sampler engagement curve flip at each tier.
• Comment provenance. Authenticated users only (Twitch login required), or anonymous + display name (a sportsbook live blog)? The auth tier sizing changes by 10×.
• Score-update provenance. Operator-typed (a live-blog editor at ESPN.com), licensed feed (Sportradar/Genius), or both? Determines whether the score-feed is kind: external (third-party SLA) or an internal service (we own the venue scouts).
• Latency budgets, per audience. The same architecture serves a 100-viewer hobbyist stream AND a 5M Super Bowl room. The SLOs are DIFFERENT — 400ms p99 for small, 2s p99 for mega — and the design must make that explicit, not aspirational.
• WS vs SSE vs long-poll. Pure web (broad ladder needed), pure native mobile (almost always WS), or both? Determines whether you build the fallback infrastructure or skip it.
• Moderation policy. Pre-moderate (synchronous classifier in the publish path, ~80ms p99 budget), post-moderate (publish first, retract on classifier disagreement), or hybrid? High-risk rooms (children's content, EU regulated, sponsored brand-safety) typically require fail-closed pre-moderation; general chat fails open post-moderate.
• Replay window. Last 30s ("welcome to the conversation"), last 5 minutes ("catch up the last segment"), or full transcript? Drives Redis Streams MAXLEN ~.
• Approximate-by-design opt-in. Do creators / leagues get to opt OUT of sampling (a regulated betting feed must deliver every score event to every connected viewer)? If yes, that lane needs to be isolated cost-wise — a 100-viewer regulated feed shouldn't be sampled, but a 5M-viewer entertainment stream HAS to be.

• Publish a comment over a persistent socket (WS primary, SSE fallback, long-poll legacy) and have it delivered to viewers in the same room.
• Subscribe to a room and receive comments + score events as they happen, with Last-Event-ID resume after a network blip.
• Receive score events for the subscribed room on a privileged lane that bypasses moderation latency and sampler drop (low-volume, authoritative).
• Approximate-by-design fanout in mega-rooms: per-viewer rate capped at a human-readable ceiling (~5 msg/s), sampled fairly by window + ranked by engagement, score-events never dropped.
• Honor slow-mode and bans: room-scoped slow-mode delays (3s, 30s, 5min) and per-user bans applied at ingest; client receives rate_limited control frame.
• Tombstone-and-retract a comment: post-moderate flow can retract a previously-delivered comment; client UI replaces it in place.
• Render last-revision-wins for score-event corrections (VAR rollback, scout typo). Revision N+1 always takes priority.
• Operator slow-mode toggle without client app reload: control-plane → room-shard → ws-gateway propagates as a server-pushed config frame.
• Cold-load history: client opens a room, fetches the last N comments via snapshot-api (CDN-cached) before opening the WS.
• Regional failover in under 60s without dual-active publishing or replay-buffer divergence; clients see feed_paused then auto-reconnect.

• End-to-end latency (writer's keypress → other viewers' screens):
• Small room (≤ 1K conc): p50 < 150ms, p99 < 400ms, p99.9 < 800ms. Conversational — must feel synchronous.
• Normal room (10K conc): p50 < 200ms, p99 < 500ms, p99.9 < 1.2s.
• Mega room (1M+ conc, sampled): p50 < 800ms, p99 < 2s, p99.9 < 5s. The viewer cannot perceive 50K msg/s; perceptual freshness matters, not delivery freshness.
• Score event (any room): p50 < 150ms, p99 < 400ms, p99.9 < 800ms — always. Score events are authoritative, low-volume, never sampled.
• Availability on the subscribe path: 99.95% monthly (21.6 min budget). Asymmetric tail: stale-comments = warn; wrong score broadcast = SEV-1 with error budget = 0; toxic content delivered to a children's room = SEV-1.
• Durability:
• Moderation ledger (moderation-ledger spine topic): retained on the spine's tiered storage (14d hot / 90d cold) as the post-moderation audit trail. Tombstone (not hard-delete) on user GDPR delete.
• Spine event log: 14 days hot tier (KIP-405), acks=all, min.insync.replicas=2.
• Replay buffer (Redis Streams): best-effort, 5-min window, lost on shard failure is acceptable (clients fall back to snapshot-api).
• Throughput at peak (reconciled):
• Mega-room (5M concurrent): 250K msg/s ingest from 5M viewers × 0.05 publish rate. Sampled fanout: 25M msg/s at the egress (5/s/viewer × 5M).
• Across 50K simultaneously live rooms (long tail, 100–10K subs each): ingest ~250K msg/s aggregate, fanout dominated by mid-size rooms, NOT the megas.
• Score-event QPS (all rooms): ~200 events/s (1.5 events/s/match × ~120 concurrent live games + admin-typed events).
• Egress (reconciled):
• Mega-room sampled egress: 25M msg/s × 250 B = 6.25 GB/s ≈ 50 Gbps for the marquee event alone.
• All-rooms steady-state global egress at peak: 80 Gbps across 4 regions.
• Olympics-gold-medal-moment burst: 130 Gbps peak; pre-warmed, not HPA-reactive (HPA is too slow for WS reconnects).
• Fanout factor flip: at room size ≥ 50K subscribers, raw S × S × r fanout exceeds phone-radio bandwidth (LTE sustained ~5–10 Mbps); sampler engages.
• Reconnect-storm resilience: survive a ws-gateway pod restart that dumps 1M sockets in 60s without auth-tier cascade.
• Multi-region: active-active with single-writer-per-room via etcd lease. Reads are active-active; writes for a given roomId route deterministically to home region. RTO 5s on shard failure; RTO 60s on full regional failover. RPO is split into two distinct boundaries that the canonical previously conflated: (1) writer-durability RPO = 0 within the home-region — a comment is acknowledged to the writer ONLY after acks=all, min.isr=2 on the home-region spine; loss of an acknowledged comment is SEV-1. (2) peer-region reader-freshness RPO ≤ MM2 lag (typical 2-5s, stress tail 60s) — peer-region readers may lag MM2 replication; on regional failover we surface feed_paused then snapshot-required fallback rather than silently serving stale or claiming a 5s SLO we can't honor under stress. Score events use RPO 0 on the reader path via provider replay (Sportradar UOF 72h, Genius 24h) covering the MM2 gap.
• Security: TLS 1.3 everywhere; short-lived RS256 JWT + signed JWKS pin (24h offline tolerance); mTLS internal; geographic blackout enforcement fail-CLOSED.

Inputs (defaults — tune per event):

• Peak concurrent in mega-room: 5M (Super Bowl / WC final / MrBeast). 30M+ exists (IPL) and triggers further sub-sharding.
• Avg publish rate per viewer (steady): 0.05 msg/s (= one comment every 20s, mega-room avg; spikes 10× during goal moments).
• Avg comment size: 250 B on the wire (JSON over WS + signature + user-id).
• Per-viewer sampled fanout ceiling: 5 msg/s (the "human-readable" rate — beyond which the eye glazes).
• Concurrent live rooms at peak: 50K (long tail of 100–10K subs each dominates broker partition decisions).
• Replay buffer per room: 1000 events (~5 min at typical rate).

Derivations — every step shown:

1. Mega-room ingest QPS: 5M × 0.05 = 250K msg/s. This is the LOAD on the comment-ingest tier + moderation.

2. Mega-room raw fanout (if NOT sampled): S × S × r = 5M × 5M × 0.05 = 1.25 × 10¹² msg/s. Infeasible on any topology — phone radios cap at ~50 KB/s (=200 msg/s) for chat. This is the math that justifies the sampler.

3. Mega-room sampled fanout: humanReadableRate × S = 5 × 5M = 25M msg/s. Per-viewer bandwidth: 5 × 250 B = 1.25 KB/s. Fits a 3G connection.

4. Mega-room sampler keep-fraction: 5 / 250000 = 2 × 10⁻⁵ = 0.002%. The sampler drops 99.998% of mega-room comments. Each viewer gets a different 5/s ranked by reservoir + engagement.

5. Mega-room egress: 25M msg/s × 250 B × 8 = 50 Gbps for the single marquee event. Across 4 regions: ~12.5 Gbps per region NIC — well within 25 Gbps NIC capacity. Without sampling, this would be 2.5 Pb/s — the sampler is a 50000× egress cost reduction.

6. Long-tail aggregate ingest: 50K rooms × 100 avg-subs × 0.05 = 250K msg/s aggregate ingest from the long tail (10M global concurrent, less the 5M in the mega-room, leaves ~5M long-tail viewers across ~50K rooms ≈ 100 avg-subs/room). Across 1024 spine partitions: ~250 msg/s/partition average. The mega-room (250K msg/s) is spread across ~100 sub-shard partitions via the composite key (room_id, hash(user_id) % 100) — per-room ordering is reconstructed at room-shard's in-pod sub-shard reducer using (ingest_ts, ingest_region) before fanout. Per-partition load = 250K / 100 = ~2.5K msg/s/partition for the mega-room slot, well below the 50 MB/s/partition (~200K msg/s at 250 B) Kafka p99-produce-latency floor. If we did NOT sub-shard, 250K msg/s on a single partition would exceed that floor and saturate one broker.

7. WS-gateway pod count: at 100K WS/pod operational (40% headroom on 250K theoretical ceiling), for global peak 10M concurrent: 10M / 100K = 100 pods minimum. Run 150 for N+1 + zonal redundancy across 3 AZs. Per-pod sysctl tuning: fs.nr_open=20M, fs.file-max=12M, somaxconn=65535, ip_local_port_range=1024-65535.

8. Mega-room sub-sharding: above 100K subscribers in a single room, a single room-shard actor's egress NIC and CPU saturate. Sub-shard by (roomId, hash(userId) % subShardCount) where subShardCount = ceil(subs / 50K). For 5M: 5M / 50K = 100 sub-shards. Each sub-shard fans out 5/s × 50K = 250K msg/s × 250 B = 62.5 MB/s — well within 25 Gbps NIC.

9. Spine partition count = 1024 with partition = hash(roomId) % 1024. Hash-routed shared topics — NOT topic-per-room. Past ~50K active rooms, Kafka controller metadata thrashes (~200K topic limit even on KRaft); shared topics scale to millions of rooms. In-pod demultiplex on consume.

10. Replay buffer footprint: per-room 1000 events × 250 B = 250 KB. 50K rooms × 250 KB = 12.5 GB hot; trivial in Redis Cluster. MAXLEN ~ 1000 (approximate trim — O(1) per XADD instead of O(log N) for exact).

11. etcd lease store: one key per active room × ~256 B/key × 50K = 12.8 MB. Comfortably below etcd's recommended 8 GB ceiling.

Architectural flip-points (with the rejected alternative + the citation):

• At 250K WS/pod Nginx+Lua tops out — flip to Centrifugo (Go) or Cloudflare Durable Objects + WS Hibernation. WhatsApp's 2M/box was Erlang on FreeBSD bare-metal; k8s + TLS terminates lower.
• At 50K subscribers per room the single-actor fanout (Erlang/Go) saturates — flip to Manifold sub-shards + relay tree (Discord 2017 Manifold blog + 2024 Maxjourney post).
• At 50K active rooms topic-per-room Kafka thrashes the controller — flip to hash-routed shared topics (room.events.{0..1023}).
• At 2M global concurrent single-region active-passive breaks under RTT — flip to multi-region active-active with single-writer-per-room fencing.
• At 50% sampler keep-fraction the "per-viewer 5/s" ceiling stops being the binding constraint and ingest QPS becomes the floor — sampler can be configured to relax above this point.

# Subscribe + bidirectional comment exchange via WSS (the dominant entry point)
GET /v1/rooms/{roomId}/stream
Host: live.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: ...
Authorization: Bearer <15-min JWT>
Last-Event-ID: 78145327                       # optional, reconnect resume

101 Switching Protocols
Set-Cookie: edge_session=<signed cookie>; SameSite=Strict; Secure; HttpOnly

After the WS upgrade, the client SDK sends and receives JSON frames:

// Client → server (publish a comment; idempotent via clientMsgId)
{ "op": "publish", "client_msg_id": "abc123",
  "room_id": "twitch:mrbeast:2026-05-17",
  "body": "BRO IS LIVE", "reply_to": null }

// Server → client (provisional ack — reconcile the optimistic render on client_msg_id;
//                   the authoritative event_id arrives on the fanout copy, below)
{ "type": "publish_ack", "client_msg_id": "abc123", "partition": 512, "offset": 88991234 }

// Server → client (comment from another viewer)
{ "type": "comment", "event_id": 88991235, "room_id": "...",
  "user": { "id": "u_q9z", "display": "...", "badges": ["sub"] },
  "body": "no way", "ts": "2026-05-17T20:14:33.122Z" }

// Server → client (score event — privileged lane, never sampled)
{ "type": "score_event", "event_id": 88991240, "room_id": "...",
  "revision": 1, "kind": "goal", "score": { "home": 1, "away": 0 },
  "minute": "78:14" }

// Server → client (retraction — post-moderate disagreement)
{ "type": "retract", "event_id": 88991235, "reason": "policy.toxicity" }

// Server → client (slow-mode engaged — config push)
{ "type": "config", "slow_mode_seconds": 30, "until_event_id": 88991500 }

// Server → client (sampler engaged — UI badge "sampled view")
{ "type": "config", "sampled": true, "keep_fraction": 0.00002 }

// Server → client (feed paused — region failover or maintenance)
{ "type": "feed_paused", "reason": "regional_failover",
  "snapshot_url": "/v1/rooms/.../snapshot", "retry_after_seconds": 5 }

// Server → client (rate-limited — caller exceeded slow-mode or per-IP cap)
{ "type": "rate_limited", "retry_after_ms": 30000, "reason": "slow_mode" }

# Cold-load snapshot (CDN edge-worker authenticates + geo-derives region BEFORE cache lookup)
GET /v1/rooms/{roomId}/snapshot
Authorization: Bearer <15-min JWT>

200 OK
Cache-Control: private, max-age=1, stale-while-revalidate=4   # edge worker caches internally, keyed on edge-derived (room, region, auth-tier); never public
{ "room_id": "...", "as_of_event_id": 88991500,
  "current_score": { "home": 1, "away": 0 },
  "recent_comments": [...last 50 post-moderated, with sampled=true marker if mega...] }

# SSE fallback (when WS Upgrade is blocked by corporate proxy)
GET /v1/rooms/{roomId}/sse
Authorization: Bearer <15-min JWT>
Accept: text/event-stream
Last-Event-ID: 78145327

200 OK
Content-Type: text/event-stream
X-Accel-Buffering: no                         # CRITICAL — disables nginx/proxy chunked buffering

event: comment
id: 88991235
data: { ... same payload as WS ... }

: keepalive                                    # comment line every 15s, beats every middlebox idle timeout

# Long-poll legacy fallback (caps concurrency, last resort)
GET /v1/rooms/{roomId}/poll?since=78145327&timeout=25
Authorization: Bearer <15-min JWT>

200 OK
# next_since = max(event_id) in events (echo the request `since` on an empty batch);
# it is monotonically non-decreasing vs the request `since`. The client always feeds
# next_since back as `since` and NEVER sends a `since` lower than the previous next_since.
{ "events": [...last event_id 88991389...], "next_since": 88991389, "next_timeout_ms": 25000 }

# Operator slow-mode (control-plane, RBAC-gated, audit-logged)
POST /admin/rooms/{roomId}/slow-mode
Authorization: Bearer <operator JWT, role: room-mod>
{ "seconds": 30, "duration_ms": 600000, "justification": "raid in progress" }

# Score-event ingress (privileged write lane — Sportradar/Genius/operator)
# Bypasses moderation AND the sampler. Idempotent per provider on (matchId, eventId, provider);
# cross-provider take-faster dedups on the normalized (matchId, kind, minute) key — provider
# eventIds are namespaced (sr:goal:5512) and never match across providers.
POST /v1/score-event
X-Provider-Id: sportradar                       # provider identity (mTLS client cert MUST match)
X-Signature: HMAC-SHA256(provider_secret, body) # forged event from a non-provider source is rejected
Content-Type: application/json
{ "matchId": "epl:LIV-ARS:2026-05-17", "eventId": "sr:goal:5512",
  "provider": "sportradar", "revision": 1,
  "kind": "goal", "score": { "home": 1, "away": 0 }, "minute": "78:14" }

202 Accepted                                     # queued to spine; fanout to every viewer, every revision
# 200 OK     — idempotent replay of a known (matchId, eventId, provider); same result, not re-published
# 401        — HMAC signature or mTLS client-cert identity mismatch
# 409        — duplicate (matchId, eventId, provider) with a conflicting payload
# 409 / 422  — room state != live (no live room to attach the score to)

Transport ladder rationale:

The data model is deliberately sparse — the system is a fanout fabric with a post-moderation audit trail on the spine, not a database. The load-bearing stores:

1. Room catalog (Postgres HA):

Plus tables: banned_users (room_id, user_id, until), pinned_messages (room_id, event_id, until).

2. Comment event (spine wire format — Protobuf):

message Event {
  uint64 event_id = 1;          // room-global total order, assigned by the room-shard
                                // reducer (see below); ingest does NOT set it — publish_ack
                                // carries a provisional spine (partition, offset) instead
  string room_id = 2;           // hash-routed to spine partition
  string client_msg_id = 3;     // user-supplied idempotency key
  oneof body {
    Comment comment = 10;
    ScoreEvent score = 11;
    Retract retract = 12;
    ConfigChange config = 13;
  }
  uint32 revision = 14;         // bumped on correction; client renders last-revision-wins
  google.protobuf.Timestamp ingest_ts = 15;
  string ingest_region = 16;    // for cross-region observability
  ModerationDecision moderation = 17;  // null on pre-moderate-fail (event suppressed)
}

Idempotency key for ingest dedupe: (room_id, client_msg_id, user_id). The same key submitted twice (browser retry, network blip) returns the same ack and never publishes a second event.

Who assigns event_id. Stateless, sub-sharded ingest (80 pods, mega-rooms spread across ~100 sub-shard partitions by (room_id, hash(user_id) % N)) cannot produce a single per-room counter — nor even a per-(room, sub_shard) one, since any of the 80 pods can produce for any user and there is no shared sequence allocator at that tier. The only monotonic per-partition sequence that exists at produce time is the Kafka offset. So ingest returns the publish_ack synchronously carrying client_msg_id plus the provisional spine coordinate (partition, offset) from the produce response — NOT a room-global id, which does not exist yet. The room-shard reducer (single-writer-per-room, holding the lease) then merges its sub-shard partitions into the room-global total order — ordering on (ingest_ts, ingest_region), the produce offset breaking ties within a sub-shard — and assigns the monotonic room-global event_id used in replay (XADD) and fanout. The client learns that authoritative id from the delivered fanout copy, reconciles its optimistic render on client_msg_id, and persists ONLY the room-shard-assigned id as Last-Event-ID — only that id is monotonic per room and safe to resume against.

3. Replay stream (Redis Streams): replay:{room_id} → XADD * <protobuf-encoded-event>, trimmed MAXLEN ~ 1000. Approximate trim (~) is O(1) per XADD; exact trim would serialize with writes and cap a hot room at ~10K writes/s. Per-key TTL = replay_window_seconds from catalog (default 300s).

4. Lease (etcd): /leases/room/{room_id} → { home_region, fencing_token, lease_ttl_30s }. Raft-linearizable. Touched on every region failover; otherwise idle.

Architecture summary — publish flow, fanout flow, score flow, cold-connect, reconnect, control plane:

Publish flow (the user posts a comment):

1. client holds a single open WSS to ws-gateway (or SSE/long-poll fallback). Sends {op:"publish", client_msg_id, room_id, body}.
2. ws-gateway authenticates the JWT against its local JWKS cache (signed-pin tolerates 24h auth outage), looks up the user's per-room ban + slow-mode state in catalog's rate-limit cache (in-process LRU, 5s TTL), and forwards the publish to comment-ingest over gRPC.
3. comment-ingest enforces rate-limit (per-user, per-IP, per-room), dedupes on (room_id, client_msg_id, user_id), calls moderation synchronously for a classifier decision (<80ms p99 budget; circuit-breaker opens on 50% errors / 30s). On block, returns moderation_failed to client. On allow or soft-flag, publishes the event to spine with acks=all, enable.idempotence=true, then writes the moderation decision to the spine moderation-ledger topic (the post-moderation audit trail, retained on the spine's tiered storage).
4. A publish_ack flows back to the client via the same WS socket, carrying client_msg_id + a provisional spine (partition, offset) (the authoritative room-global event_id arrives with the fanout copy — step 9). The publisher SEES THEIR OWN COMMENT in the local UI immediately on ack (optimistic render), reconciled on client_msg_id when the fanout copy arrives.

Fanout flow (others receive the comment):

1. spine is partitioned by hash(roomId) % 1024. Each partition is consumed by one room-shard actor that owns the room (or sub-shard, for rooms > 50K subs).
2. room-shard first checks lease to confirm it still owns the single-writer fencing token for this room (fail-CLOSED on lease loss → emits feed_paused to subscribers and makes NO further writes), and only THEN appends the event to replay (XADD replay:{roomId} with MAXLEN ~ 1000, carrying the fencing token so a stale writer's append is rejected at the store).
3. room-shard decides: if room.size > 50K and sampling_policy != "none", push the event to sampler; else push directly to ws-gateway fanout.
4. sampler (only on mega-rooms) runs a 200ms reservoir window + engagement-rank + sticky-thread scoring, emits a per-subscriber fanout list at ≤5 msg/s/viewer. Score events bypass the sampler entirely on the privileged lane.
5. ws-gateway flushes the WS frame into the open socket of every subscribed client (per-socket coalescing on backpressure — coalesce only same-(roomId, eventId, revision), never collapse a revision N+1).
6. client renders the comment, persisting last_event_id to disk for reconnect.

Score flow (admin or licensed feed):

1. score-feed (external — Sportradar UOF / Genius / live-blog editor's PUT) calls comment-ingest on the privileged HTTPS endpoint POST /v1/score-event. Auth via mTLS + signed payload.
2. comment-ingest validates against catalog (matchId in live state), publishes to spine. Score events have body.score and bypass the moderation classifier (operator-typed is trusted; licensed feed is contractually accurate).
3. Fanout flow steps 5–10 apply, with one difference: sampler is skipped on body.score events. Every connected viewer receives every score event, every revision.

Cold-connect / subscribe flow:

1. client resolves live.example.com via gslb (DNS TTL 30s) to the nearest healthy region's anycast prefix.
2. client opens WSS to anycast-lb; admission control runs at L4 (1.2× steady-state token bucket, drainable by control-plane).
3. anycast-lb forwards raw TCP to a co-located ws-gateway instance.
4. ws-gateway terminates TLS + WS Upgrade, validates the JWT against local JWKS cache + signed pin, reads catalog for room entitlement + blackout + slow-mode (fail-CLOSED with coordinated Retry-After).
5. ws-gateway registers the subscription on the correct room-shard (via gRPC op: control register edge).
6. If client supplied Last-Event-ID, ws-gateway does XRANGE on replay for the gap; on gap > buffer head, returns snapshot-required and the client fetches snapshot-api (CDN-fronted).

Reconnect + replay flow (the dominant resume path — every tunnel, every WiFi handoff):

1. Client supplies Last-Event-ID: X on reconnect. ws-gateway runs XRANGE replay:{roomId} (X + against replay, replays each gap event into the socket, only THEN subscribes to live. Branch: replay.

Snapshot fallback (gap > replay buffer or first-load):

1. client GETs /v1/rooms/{roomId}/snapshot against snapshot-api (CDN-fronted with 1-5s TTL — absorbs 90%+ of cold loads).
2. snapshot-api reads catalog for room state, replay for the last 50 events (XREVRANGE), and the deeper replay tail for older comments if requested, assembles in <30ms p99. Branch: snapshot-required.

Control plane (operator actions, failover, slow-mode):

1. Operator hits control-plane /admin/rooms/.../slow-mode or /admin/region/failover. RBAC-gated (room-mod vs region-ops roles), audit-logged.
2. Control-plane writes the new state to catalog AND publishes a control event to spine. Room-shard consumes, pushes a config frame to subscribers via ws-gateway. For failover, control-plane wraps drain → lease-CAS → DNS flip in one transaction (single command, audited).

Observability:

1. ws-gateway and room-shard emit OTel spans to tracing (tail-sampled, with always-sample-on-incident override).

Multi-region: active-active, single-writer-per-room by consistent-hash home_region(roomId). Reads from any region. MirrorMaker 2 replicates spine cross-region for read-side fanout (NOT for redundant writes). etcd lease coordinator spans 3 regions for quorum.

1. The MrBeast room — approximate-by-design is not a degradation, it's a feature.

A 5M-concurrent room with 0.05 msg/s/viewer publish rate ingests 250,000 comments per second. A 3G phone radio sustains ~50 KB/s on the downlink. At 250 B/comment that's 200 msg/s before the buffer fills — and the eye glazes over above ~5 msg/s anyway. The sampler is therefore not an emergency degradation; it is the product surface of a mega-room.

The policy lives in three layers:

• Fairness: 200ms reservoir window. Within each window, every incoming comment has equal probability of being kept, regardless of submission timestamp inside the window. This avoids the "early bird wins" bias of a naive head-cut.
• Value: engagement-rank scoring within the window — comments with replies, reactions, sponsored badges, moderator pins, or from accounts the viewer follows get a multiplicative bonus. The actual scoring is score = base + reply_count × 5 + reaction_count × 2 + badge_bonus + sticky_thread_bonus.
• Personalization: sticky-thread bonus — if a viewer has replied to a thread within the last 5 minutes, comments in that thread are 4× more likely to land in their sampled view. Costs CPU on the sampler per-viewer.

The load signal driving sampler engagement is two-source AND-gated: queue_depth > threshold AND cpu > threshold. Single-signal sampling has the classic Spectroscope failure mode — a CPU blip from a GC pause flips the sampler to "drop 90%" during a non-spiky moment and the room goes silent for ten minutes, viewers think it's broken. AND-logic means both signals must agree; alert fires if sampler engages outside expected windows. Floor at 50% drop — never sample more aggressively than that without operator override, so a sampler bug doesn't take the room completely dark.

Score events are tagged in the protobuf body and bypass the sampler entirely. The privileged-lane invariant is what makes the "goal at minute 78" never get sampled out — same as the live-sports-scores canonical, but here it coexists with a 99.998%-drop comment lane.

2. Topic-per-room breaks at 50K rooms — hash-routed shared topics are the answer.

The naive design: one Kafka topic per roomId. Easy mental model, lets consumers subscribe room-by-room. It breaks before you fill broker NICs because the Kafka controller's topic-metadata snapshot grows linearly with topic count, and controller failover time grows with snapshot size. Past ~200K topics (KRaft pushes higher than ZooKeeper, but it's still finite), controller failover takes 30s+ during which producers stall and partitions can't rebalance. We hit this at 50K active rooms, far below the broker-CPU or NIC ceiling.

The fix is hash-routed shared topics: 1024 partitions across 4 topics (room.events.{0..3}.{0..255}), with partition = hash(roomId) % 1024. Producers compute the partition deterministically; consumers (room-shards) own a partition and in-pod demultiplex the partition stream by roomId. This is YouTube Live Chat's pattern, and Slack's. The cost: a consumer can no longer "subscribe to just the rooms it cares about" — it gets the whole partition. The benefit: scale to millions of rooms without controller pain.

Per-partition load math: at 250K msg/s peak for a mega-room landing on ~100 partitions (one room sub-sharded, ~2.5K msg/s each), and a ~250 msg/s/partition baseline for the long tail, p99 produce latency stays well under 50ms (Kafka's well-known LinkedIn floor of ~50 MB/s/partition = ~200K msg/s at 250 B).

3. Single-writer-per-room is the load-bearing invariant that makes active-active honest.

The trap in any active-active fanout system is the "dual-write for safety" anti-pattern. If a comment for room=R is accepted in us-east-1 AND eu-west-1 simultaneously, the replay buffer in each region holds different orderings, and on cross-region read the user sees their own comment twice, or interleaved with itself, or in inverted causal order with a reply. This is the shape of Slack's 2018 message-ordering bug and the Jepsen-documented MongoDB / CockroachDB orderings.

Our defense: single-writer-per-room via etcd lease. Every roomId deterministically hashes to a home_region. Comment-ingest in the OTHER region (the reader-region) PROXIES the publish via mTLS+gRPC to the home-region's comment-ingest — it does NOT publish locally. Reads (consume from spine + push to ws-gateway) are active-active everywhere via MirrorMaker 2. Writes go to ONE region only.

The lease is what makes this safe under failover: when the operator runs score-ops region failover, the lease CAS in etcd atomically moves home_region from us-east-1 → eu-west-1. Old-region ingest sees lease-loss within 5s, fails CLOSED on local publishes (returns feed_paused-style 503 with Retry-After), forwards in-flight publishes to the new region. The drain order (drain L4 → CAS lease → flip DNS) is the same pattern that's spelled out in the live-sports-scores canonical and that prevents the Cloudflare Jun 21 2022 / Slack Jan 4 2021 dual-active footgun.

4. WS-SSE-long-poll fallback ladder — different infra per rung or you co-locate the failure.

The client SDK climbs the transport ladder: WS first, SSE second, long-poll last. The trap is putting all three rungs behind the same CDN, same anycast prefix, same edge — when the edge fails, every rung fails together. We split:

• WS terminates at our own anycast-lb + ws-gateway (Centrifugo / Durable Object).
• SSE terminates at Fastly Fanout (or our equivalent GRIP proxy), which converts the SSE subscription into a plain HTTP GET against ws-gateway — origin sends a Grip-Hold instruction and drops; Fastly holds the SSE client connection at the edge until it has something to push. Critical headers: X-Accel-Buffering: no + a 15s keepalive comment line. Without these, corporate proxies and some nginx defaults buffer 8 KB of chunked output and SSE arrives in bursts.
• Long-poll terminates at a third path — a separate CDN with a 25s server-side hold. Concurrency-capped on the client SDK to avoid quadratic load if the user opens 12 tabs.

The client SDK enforces a bounded climb: at most 3 rungs, with decorrelated-jitter backoff between rungs. On rung-3 failure, surface "Connection problems — last comment delivered at HH:MM" rather than recurse. Telemetry: transport=ws/sse/longpoll is a first-class label on every connect metric.

Mobile is the inversion: native apps almost always prefer WS (single multiplexed socket; opening a new TCP/TLS connection wakes the radio for ~10–20s of high-power state and burns battery).

HTTP/2 SSE has its own trap: multiple SSE streams share a TCP connection, a lost packet stalls every multiplexed stream (TCP HOL blocking that H2 multiplexing was supposed to fix but only fixed at the app layer). HTTP/3/QUIC isolates loss per QUIC stream and is the real fix, but at our scale we still see H2 in legacy regions — sharding SSE across sse-1.live.example.com, sse-2.live.example.com works around it.

5. Moderation in the publish path without putting it ON the publish-path SLO.

Moderation is synchronous in the default policy — comment-ingest waits for the classifier verdict before publishing to spine. That coupling is a 3 AM page waiting to happen: classifier p99 climbs from 30ms to 1.2s under load and the publish-path latency follows; users retry; the queue compounds.

The bulkhead policy is per-room, set in the catalog moderation_policy field:

• pre-moderate-strict — children's rooms, regulated content, sponsored brand-safety. Fail-CLOSED on classifier timeout: comment is HELD, user sees moderation_pending with retry. Classifier outage = comments stop. This is the correct behavior for these rooms.
• pre-moderate (default) — general chat. 80ms p99 budget. Circuit-breaker opens at 50% error rate / 30s; while open, comments fall through with flagged=pending, async classifier runs over them with retroactive retract on disagreement.
• post-moderate-fast — mega-stream chat. Comments publish immediately with mod=async; classifier runs over the spine moderation-pending topic; on block verdict a retract event is published to the room and viewers see the comment replaced in place. The trade is that toxic content has a ~500ms window where it's visible.

The single critical invariant: the classifier never extends the per-comment publish-path latency more than the budget. If it can't decide in 80ms, fall through per policy. Don't queue the user's send button waiting for the classifier.

6. Replay buffer hot-key — the 90%-of-reconnect-users-hit-one-key problem.

When a region drops and 1M clients reconnect simultaneously to a mega-room, every one of them executes XRANGE replay:{roomId} (last_event_id +. That's one Redis key, on one shard, taking 1M reads in 60s. CPU on that Redis node saturates; XREAD p99 climbs into seconds; the reconnect spins.

Defenses, layered:

• Per-room read-replicas: replay:{roomId} is replicated to 2 read-replicas (Redis Cluster with replica-read mode); ws-gateway picks one per reconnect via consistent-hash on userId. Spreads the read load 3×.
• Edge ring-buffer: ws-gateway maintains an in-pod ring buffer of the last 30s of events for every room it serves a subscriber for. On reconnect with Last-Event-ID within that window, serve locally — never hits Redis.
• Snapshot fallback at gap > buffer: if Last-Event-ID < replay buffer head, return snapshot-required and route to snapshot-api (CDN-cached). Most reconnects after long tunnels (> 5 min) bypass replay entirely.
• Reconnect-fanout token-bucket at anycast-lb: the L4 admission cap is 1.2× steady-state, deliberately undersized for storms. Storms spread across the decorrelated-jitter window naturally.

What we accepted, with the rejected alternative and the citation:

Deliberately scoped out (v1): mobile push (APNs/FCM), the approximate live-viewer-count badge, and the delivery-telemetry / long-retention comment archive are out of scope for this canonical, which focuses on the core publish → moderate → fanout spine. Each layers back on as a new consumer over the same Kafka spine — a push-dispatcher consumer group off a notif.push topic, a presence-count reducer fed by ws-gateway connect/disconnect events, and a ClickHouse/S3 archive consumer off moderation-ledger — without touching the core path.