Reddit / Hacker News — a worked solution

What each component is for

Client

Renders ranked listings (front page, /r/:sub/hot, /comments/:id), opens long-running websockets for live comments, and POSTs votes/submits with an Idempotency-Key header. Browsers receive HTML on cold loads + JSON on subsequent paginations; mobile is JSON-only.

Why it exists. Reddit-shape products are read-tail-dominated — a session typically loads 5+ listings before a single vote — so the client is where the SLO budget lives. We picked HTML+JSON over GraphQL precisely because edge-cacheable URLs (one URL = one ranked listing) are the load-bearing primitive that lets Fastly absorb 95% of read QPS.

When it fails. If the client retries non-idempotent votes after a 408/5xx — and they will, on every cellular handoff — vote-double-counting visibly skews ranking. Detection is the vote-audit job that compares raw vote-row count to score deltas; mitigation is the server-side idempotency dedup keyed by (account, target, vote_value) with a 24h TTL.

CDNFastly

Fronts every listing fetch (/r/:sub/hot.json, /comments/:id.json) and signed media URLs. Caches listing JSON for 10s with surrogate-key purge keyed on (sub, sort), so the ranker can soft-purge a listing in <1s after each ZADD. Signed URLs gate any non-public sub.

Why it exists. At ~32K listing QPS peak with 40% landing on the front page (one logical key), origin would melt without short-TTL edge caching. We considered cache-asiding only at the gateway and skipping the CDN, but origin's listing-read tier still gets crushed by the front-page hot key — only the edge gives geographic spread that absorbs viral traffic. Fastly's surrogate-key model gives us the freshness invariant a time-based TTL alone cannot.

When it fails. Edge cache poisoning: a stale 'hot' object pinned at one POP for 60s while the actual hot listing has rotated away. Detection is the synthetic prober (hosted on the monitor) that asserts Age on /hot is <30s; mitigation is the surrogate-key purge fired on every ranker emit, plus a hard 30s ceiling on edge-ttl. Loss of CDN entirely would push origin to ~10× current load — the runbook is to flip to a secondary CDN provider via the LB's backup origin pool, accepting a 90s propagation gap.

Load Balancer · Anycast EdgeAWS NLB + BGP anycast

Terminates anycast traffic from the CDN miss path, runs L7 health checks against the gateway tier (200ms timeout, 2-of-3 threshold), and round-robins requests across healthy gateway pods cross-AZ. On region failover, anycast withdraws the unhealthy region's prefix and traffic re-routes within ~30s — strictly faster than DNS-based failover by avoiding TTL propagation.

Why it exists. Region failover via DNS depends on resolver TTL respect — production resolvers cache aggressively past TTL, so the 'half users on dead region' pattern (failure mode 19 in the SRE Workbook) hits exactly here. We considered a 30s DNS TTL on regional records but rejected it because (a) DNS amplification spikes resolver load, and (b) anycast withdrawal is sub-30s without any client-side caching games. ECMP at the edge spreads traffic uniformly; gateway health-checks remove unhealthy pods within one check interval.

When it fails. Gray failure: health checks pass but real traffic fails (gateway returns 200 on /health but 503 on /vote). The LB happily routes to broken pods. Detection: the monitor's synthetic prober exercises a full /vote round-trip every 10s and alerts on >1% failure rate. Mitigation: deep health check that touches downstreams (rate-limit-redis ping + JWKS fetch) so /health fails when the gateway is not actually serviceable.

API GatewayEnvoy + ratelimit sidecar

Terminates TLS, runs WAF rules, enforces per-IP/per-account token-bucket rate limits (60 votes/min/account, 10 submits/hour/account, 600 listing fetches/min/IP) by INCR-ing buckets in rate-limit-redis, validates session cookies or OAuth bearer (JWT verified locally against a JWKS cache, refreshed every 5min), then dispatches: /hot|/top|/new → listing-read; /comments/:id → comment-tree; POST /vote|/submit → vote-svc. Strips Idempotency-Key into a request attribute.

Why it exists. Submission-spam waves and vote-storms are abuse problems, not capacity problems — they need to be shed AT THE EDGE before they consume any backend capacity. The gateway is also where karma-farm shadow-banning shunts traffic to a low-priority queue (the ranker still sees the votes, but tags them suspect). Doing rate-limiting in each backend service would mean every backend grows its own Redis client and rules diverge.

When it fails. If rate-limit-redis is partitioned, gateway must fail-open or fail-closed. Fail-closed = real users see 429s during a Redis blip; fail-open = a spam wave punches through. We fail-OPEN with a circuit-breaker that re-engages on Redis recovery, AND the fraud-scrubber re-evaluates the burst window after the fact — the spam still gets shadow-removed, just not blocked at the door. Detection is the ratelimit-redis p99 + fail-open-rate gauge. Second failure: JWKS-cache staleness — if the IdP rotates and JWKS cache misses propagate, every authed request 401s for up to 5min. Detection: 401 rate spike post-JWKS-rotation; mitigation: pre-warm JWKS cache on rotation event (the IdP webhooks the gateway on every rotation).

Cache · Rate Limit BucketsRedis 7 (single shard + 1 follower)

Holds the per-(account, ip) token-bucket state for the gateway's rate-limiter. Lookup key: rl:{account|ip}:{endpoint}. Each gateway request issues a LUA-scripted INCR-and-check + EXPIRE, returns the remaining tokens; the gateway decides allow/deny. Hot keys: any IP behind a NAT (corporate / school VPNs) drives a single key to thousands of QPS.

Why it exists. Storing rate-limit state in the gateway's process memory is cheap until you scale to 12 gateway pods — a per-pod bucket lets a single bot burst ×12 by hitting different pods. We considered hashing client IP to a sticky pod (so each pod owns a deterministic IP slice) and rejected it because asymmetric load and pod failures would drop limit state. Centralized Redis is the correct trade — sub-ms p99 limit checks, shared state across all gateway pods.

When it fails. If rate-limit-redis is unreachable, gateway fails OPEN (per gw.failureMode), so a partition is non-fatal. The harder failure is bucket-key contention on a NAT'd IP — a corporate proxy with 50K users behind one IP saturates one bucket key. Detection: per-key INCR rate >10K/s alarms on the monitor. Mitigation: hash subkey on user_agent for IPs flagged as NAT'd; the fraud-scrubber labels them.

Service · Listing ReadGo (Baseplate.go)

Resolves /r/:sub/hot.json by ZRANGEing the precomputed sorted set in listing-cache, multi_getting per-thing data (title, score, thumbnail) from render-cache (with a thing-db replica fallback for cold rows), and returning the page. Front-page (/r/all, /r/popular) is a fan-in over O(50K) sub listings precomputed by the ranker into one global candidate ZSET.

Why it exists. Computing hot at read time would mean SELECT…ORDER BY hot(ups,downs,date) LIMIT 25 across millions of posts on every request — at 32K QPS peak, that is a query-planner death spiral. Precomputed Redis ZSETs reduce read-time work to O(log N) per page. We rejected Postgres materialized views because cross-sub fan-in for /r/all needs to merge 50K row groups in <50ms, and Postgres simply cannot do that. We rejected client-side ranking because half our DAU is on phones that cannot afford the CPU.

When it fails. If listing-cache misses (Redis evicts under memory pressure or a shard fails over), every concurrent request stampedes the ranker queue and thing-db replicas. Detection is listing_miss_rate >5% paged within 60s by the monitor; mitigation is single-flight (request-coalesced ranker recompute, others block on a promise) plus a 30s stale-while-revalidate served from the prior cached page. The edges to listing-cache and listing-read use circuit-breaker (not exp-backoff) for exactly this reason — at 32K QPS, exp-backoff produces a 27× retry amplification under partial Redis failure. Pi Day 2023 was this shape: bring the cache back BEFORE the front door reopens.

Service · Comment TreeGo (Baseplate.go)

Fetches /comments/:post_id by reading the precomputed preorder traversal of the comment tree from thing-db (a flat list of comment_ids with depth/parent), multi_getting per-comment HTML chunks from render-cache, and decorating each comment with the requesting user's vote state (read from vote-store via Bloom-filter negative-lookup). Trees deeper than 250 children at any level surface a MoreChildren stub; the client paginates into them via /morechildren.

Why it exists. The largest threads (e.g. /r/AskReddit megathreads) reach 50K+ comments. Building this tree from a recursive Postgres CTE on every fetch would push p99 to seconds and DOS the DB. Pre-rendered HTML fragments + a flat preorder array means a thread fetch is one tree-skeleton query plus one memcached multi_get plus one Cassandra Bloom probe per displayed comment.

When it fails. When the #1-on-front-page thread's render-cache expires, every concurrent fetcher races to recompute. With 10K concurrent viewers, that is 10K parallel CPU-bound renders. Detection: render_p99 >5s paged; mitigation is single-flight per (post_id, subtree_root) plus chunked subtree caching (so a vote on one child does not bust the whole tree). Discord's Scylla migration retro covers this exact fanout.

Service · Vote / SubmitGo (Baseplate.go)

Handles POST /vote and POST /submit. For votes: validates Idempotency-Key, runs Cassandra LWT (Paxos compare-and-set) on (user_id, target_id, idempotency_key), atomically writes (a) the vote row + (b) an outbox event into the same Cassandra LOGGED batch keyed on user_id, returns 200 with the new score read from the fast post-counter row in thing-db. Best-effort first-attempt Kafka publish runs in-line; the outbox-tailer sidecar handles catch-up. For submits: post_id = deterministic hash(idempotency_key, author_id) so retries hit the same row, INSERT into thing-db with ON CONFLICT DO NOTHING, then emit a 'new submission' outbox event.

Why it exists. Votes MUST be idempotent — clients retry on every network blip and a duplicated vote skews ranking visibly. We picked Cassandra LWT over Postgres unique-constraints because the vote-history fan-out (a viral post takes 1K votes/sec on a single key) saturates a Postgres shard. We rejected Redis-only counters because we lose the per-(user, target) audit row that the fraud-scrubber depends on. We chose the outbox pattern over 2PC across Cassandra + Kafka because 2PC over Kafka is operationally a tar pit.

When it fails. Three signals on this tier, all on the monitor: (1) LWT_failures rate spike — Cassandra cannot form a write quorum (two AZs partitioned), 503 to client, retry with same Idempotency-Key. (2) Outbox-row-age — now() - min(outbox.created_at WHERE published_at IS NULL) >30s pages on a dead outbox-tailer (the silent-killer; both db_score and ranker_score stop moving together so the score-divergence reconciliation cannot detect it). (3) Score-divergence — the reconciliation job alerts on |db_score - ranker_score| > 5%, and on alert we drain the outbox before re-rank.

Worker · RankerGo + Kafka consumer

Consumes vote events off Kafka in a per-(post_id) keyed partition. Recomputes hot = log10(max(|score|,1)) + sign(score)·t_submit/45000 for posts, where t_submit is the submission time in epoch-seconds — newer submissions earn a larger term, so recency is rewarded; Wilson-score lower-bound at 95% confidence for comments (the 'best' sort); HN-style (P-1)/(T+2)^1.8 wired in for /r/all gravity decay. Reads a 5s-TTL per-post tally cache (in-process) before fanning out to the 16-way counter-shard-split tally read on viral posts. ZADDs the new score into listing-cache for every (sub, sort) the post is in, plus the global candidate set for /r/all. Cadence: per-vote on viral posts (top 1K), per-batch every 5s on cold posts to amortize Redis ops.

Why it exists. Read-time ranking does not scale beyond ~500 listing QPS — Reddit pre-2010 melted exactly here. Async ranker keeps the vote-write path at 120ms p99 regardless of how many listings the post participates in. The fan-out (one vote → up to 4 ZADDs) happens off the user's request. We considered Kafka Streams stateful processors but rejected them — the score function is dumb arithmetic; the value is in batched coalescing, not in stream-SQL.

When it fails. Consumer lag is the silent killer — if the ranker falls behind 5+ minutes, the front page is yesterday and users disengage with no error firing. Detection: the monitor scrapes the ranker's freshness heartbeat (now() - max(listing.generated_at) >5min) and pages on staleness, not on errors — symptom-based alerting per the SRE Workbook. Mitigation: load-shed low-value events (anonymous-view signals before votes), autoscale on lag not CPU, and a kill-switch to drop fraud-suspect votes from the rank input when scrubber is also behind.

Worker · Fraud ScrubberGo + Kafka consumer + graph clustering

Tails the same vote-stream on its own consumer group. Maintains a rolling fraud-score per (account, IP, device, vote-time-correlation) using sliding-window graph clustering (1h windows, edges weighted by co-vote count). On suspicion, writes a 'scrub' record into vote-store and emits a 'recompute(post_id)' control event to the ranker — the surfaced votes remain visible to the offending voter (shadow-ban semantics, the load-bearing trick) but are excluded from rank input. Also shadow-marks accounts in thing-db so subsequent submits go to a low-priority ranker queue.

Why it exists. Vote manipulation is existential for a ranked product — without an offline scrubber, a $100/day karma farm pollutes the front page and the product dies. We chose async over inline because the fraud signals (graph clustering on co-voting, cross-account IP overlap, temporal pattern matching) need a window of votes to evaluate; inline would push vote-ack p99 past the 120ms budget. We chose shadow-removal (not blocking) because if attackers see their votes get blocked, they iterate the attack — invisibility is the moat.

When it fails. Scrubber lag pollutes the product visibly. SLO: scrubber lag <1h. If breached, runbook: (a) drop low-confidence votes from rank input until lag drains, (b) re-rank affected posts post-scrub. Detection is consumer-lag-on-fraud-topic page from the monitor; mitigation is the inline lightweight check at vote-time (gateway-level, IP velocity only) as a load-bearing fallback.

Worker · Outbox TailerGo sidecar (1-per vote-svc replica)

A sidecar (one per vote-svc replica) that reads vote-store rows where published_at IS NULL AND created_at > now() - 24h, publishes each event to vote-stream (Kafka), and updates published_at = now() on success. Runs every 100ms. Vote-svc still publishes synchronously on the happy path (e14); the tailer is the catch-up path only fires when the synchronous publish failed (Kafka blip, partition leader election, broker GC pause).

Why it exists. The synchronous publish from vote-svc is best-effort — if Kafka is briefly unavailable, the vote is durable in vote-store but absent from the ranker's input. Without the tailer, votes stuck in the outbox stay invisible to the ranker forever, the front page silently freezes for those posts. We considered a single global tailer (one process tailing all outbox rows) but rejected it as a SPOF; a sidecar-per-replica gives natural sharding by user_id partition assignment.

When it fails. If a tailer dies and no replacement spawns, votes pile up in the outbox; rank visibly freezes for those users' votes (depending on partition coverage). Detection: the monitor scrapes now() - min(outbox.created_at WHERE published_at IS NULL) >30s — the canonical's load-bearing outbox-lag signal. Mitigation: monitor's heartbeat-absence alarm plus k8s liveness probe on the sidecar. Re-publish is at-least-once; the ranker dedupes via idempotency_key.

Cache · Listing ZSETsRedis 7 (cluster mode)

Holds the precomputed hot/top/new listings as Redis sorted sets keyed listing:{sub}:{sort}. Each ZSET holds 1000 thing_ids ranked by score. Reads are ZRANGE; writes are ZADD/ZREMRANGEBYSCORE on every vote (top-of-listing posts) or ranker-batched (cold posts). The front-page ZSET is replicated across all 16 shards as listing:_all:hot:0..15 and reads round-robin across them — this is the load-bearing mitigation for the 13K front-page-read QPS hot key.

Why it exists. Listings ARE the read product — at 32K QPS peak with 40% on front-page (one logical key), Postgres replicas would saturate before reaching 5K QPS. Redis ZSET gives O(log N) updates and O(K) range reads in-memory, which is what an 80ms p99 cached budget demands. We rejected DynamoDB DAX because the per-key write rate (1K vote-ZADDs/sec on the #1 post) exceeds DAX node throughput. We rejected an in-process LRU on each listing-read replica because cross-replica score divergence is a worse failure than network hops.

When it fails. Cold restart is the page that shaped the May 2023 Reddit outage. If the cluster restarts cold, hit-rate drops below 50% and Cassandra-thing-db gets crushed. Detection: monitor scrapes Redis hit-rate; <90% pages. Mitigation: refuse cold-start traffic — admission control (at the LB) caps origin QPS to 10% of normal until hit-rate recovers, ranker pre-populates from a snapshot, the front door re-opens only after hit-rate >90%. Cross-region cold-rebuild RTO for the full ZSET set (~50K subs × 1K thing_ids) is ~10 min — this is the real RTO, not the 30s edge cutover. Hot-shard meltdown on the front-page key is mitigated by the 16-way fan-out described in keyChoices.

Cache · Render FragmentsMemcached

Holds rendered HTML fragments — per-comment HTML, per-listing-card HTML, and the user-flag overlay for vote-arrow state. Reads are mget over hundreds of keys per page; writes are issued by listing-read and comment-tree services on cold renders, and invalidated by the ranker on score changes greater than the 'visible delta' threshold (avoids invalidating on every fuzz).

Why it exists. Pre-rendering once and serving N times is what makes /r/AskReddit megathreads feasible. Rendering 50K comments per request would saturate the comment-tree CPU at <50 QPS. We chose Memcached over Redis here because Memcached's slab allocator handles variable-sized HTML fragments better than Redis's hash table, and we already use Redis for the sorted-set workload — keeping caches workload-specialized lets us tune memory and eviction independently.

When it fails. Two pages on this tier. (1) Connection-pool exhaustion: a megathread fanout drives every comment-tree replica through its per-replica Memcached connection limit (~256 conns/replica) before any infrastructure error fires. Detection: mget p99 spike with no Memcached-side error. Mitigation: connection-pool autosize on QPS, single-flight per-key. (2) Topology change: a node added/removed effectively invalidates ~1/N keys, triggering a render storm on whichever threads land on the new shard. Detection: render p99 spike. Mitigation: ketama hashing + rolling-add-then-fill (warm the new node from thing-db before exposing it to reads).

Stream · Vote EventsKafka 3.7

Single source of truth for every vote and submission event. Producer: vote-svc (synchronous best-effort) + outbox-tailer (catch-up). Consumers: ranker-worker (per-post partition), fraud-worker (separate consumer group), and a search-indexer consumer (out-of-scope for the main flow). Per-message body: (user_id, target_id, target_kind, dir, ts, idempotency_key, fraud_signals, ranker_version_pin).

Why it exists. The outbox pattern (write the vote row + the event in the same Cassandra LOGGED batch) decouples vote-write durability from ranker availability — the ranker can be down for 30 minutes and votes still durably commit; the rank just lags. We rejected synchronous gRPC fan-out from vote-svc to ranker because every additional sync hop multiplies vote-ack p99, and ranker recomputation on a viral post is itself bursty. We rejected RabbitMQ (Reddit's historical choice) because partition-keyed ordering is load-bearing and Kafka's per-partition leader election handles vote-storms more gracefully than RabbitMQ mirrored queues.

When it fails. Two leading indicators on the monitor. (1) under-replicated-partitions >0 — with min.insync.replicas=2 of RF=3, one broker loss puts every partition at the edge of write availability, and ISR shrinkage is 30+ seconds ahead of the producer 503 symptom. (2) Consumer lag on votes.events >30s p99 — the dominant pipeline failure shape. The poison-pill scenario (one bad message blocks one partition) is mitigated by max-retries-then-DLQ; DLQ runaway is bounded by per-partition consumer isolation so 1/256 of votes never blocks the rest. Loss of in-sync replicas below min.insync.replicas → producer 503s (we deliberately fail closed; back-pressure to vote-svc is preferable to acking a non-durable vote).

Monitoring · Freshness + SLOPrometheus + Alertmanager + synthetic prober

Scrapes per-service metrics every 15s — listing-cache hit-rate, ranker freshness heartbeat, vote-svc outbox-lag, vote-store coordinator queue depth, vote-stream under-replicated-partitions, render-cache mget p99, gateway 401-rate (JWKS), rate-limit-redis INCR rate per key. Hosts the synthetic prober (Go binary, every 10s) that exercises /hot, /comments, and /vote round-trips end-to-end from each region. Pages on SLO breach via PagerDuty integration.

Why it exists. Gray-failure detection — silent stale ranker, dead outbox-tailer, JWKS-rotation cascade, edge cache poisoning — has no error signal in any single service's logs. Symptom-based alerting requires a CENTRAL place to express 'now() - max(listing.generated_at) >5min' across all ranker replicas. We considered piping these signals through a log-shipper to ELK and triggering on log queries, but rejected it because ELK alert latency is minutes — the freshness probe must page within 60s.

When it fails. If the monitor itself is degraded, every other system loses its detection signal — the worst possible failure (you don't know your systems are down). Mitigation: cross-region peer health-checks (each monitor instance watches the others' heartbeats); detection: 'absence of expected heartbeats' from any monitor instance pages the on-call's pager via a parallel cross-region path.

SQL DB · Thing DBPostgres 16 + Citus

Canonical store for posts (thing_link), comments (thing_comment), accounts (thing_account), and the EAV data_<type> tables. Holds the comment-tree skeleton (ordered preorder traversal with depth/parent), post metadata (title, url, author, created_at, deleted, spam, ranker_version_emitted), and the (fuzzed) cumulative ups/downs counters used for read-time score reconciliation. The submission_idempotency table dedups POST /submit by (Idempotency-Key, author_id) with a 24h TTL.

Why it exists. Reddit's classic ThingDB-on-Postgres survives because (a) submissions and comments are write-rare (~400 writes/sec total) so single-leader-per-shard is fine, and (b) Postgres's relational guarantees on author + thread_id integrity matter for product invariants (comments cannot reference a deleted post). We rejected Cassandra here because the comment-tree skeleton query (ORDER BY preorder_idx WITHIN post_id) is exactly the range scan Cassandra does poorly without a hand-tuned partition key, AND we need transactional INSERT-and-bump-counter on submissions.

When it fails. Two pages on this tier. (1) Compaction/autovacuum storm at peak hour — write p99 spikes from 10ms to 2s and bleeds into vote-svc (which carries both submission and comment writes — e12/e30). Detection: monitor scrapes pg_stat_progress_vacuum + write p99. Mitigation: per-shard autovacuum cost-limit caps tied to peak QPS (back off at >80% peak), manual VACUUM scheduled off-peak, per-shard write-rate cap at vote-svc so back-pressure flows to the user (429) rather than to the ranker queue. (2) Sync replication backpressure — a slow follower stalls the primary's commit, write p99 climbs across the shard. Detection: pg_stat_replication.lag >50ms; mitigation: degrade the offending follower to async, alert on quorum loss.

NoSQL DB · Vote StoreCassandra 4.1 (RF=3, quorum)

The high-write side of the system. Stores (user_id, target_id, target_kind, dir, ts, idempotency_key, scrubbed_flag) — partition key user_id (a user's vote-history is one partition), clustered by target_id. A second materialized view is partition-keyed by target_id for the ranker's per-post tally. Bloom filters give sub-ms 'did user X vote on target Y?' negative answers used on every comment-tree decoration. The outbox table (outbox) is co-partitioned with the vote table on user_id so a single LOGGED batch atomically writes the vote row + the outbox event.

Why it exists. Votes are the heaviest write workload (~15K QPS peak, ~500M/day, 73TB indexed at 5yr) AND the heaviest negative-lookup read (the comment-tree decoration: 'has this user voted on each of these 100 comments?'). Cassandra was specifically chosen by Reddit in 2010-2011 over Postgres because Bloom-filter negative lookups are sub-ms at this scale. Postgres unique-index lookups, by contrast, force a B-tree probe per check — orders of magnitude slower at 5K decoration QPS × 100 keys. We considered DynamoDB; rejected because the per-partition write ceiling (~3K WCU) hits the viral-post hot key.

When it fails. Three pages on this tier. (1) Hot-partition write meltdown (#1 viral post takes 1K votes/sec): leading indicator is coordinator_queue_depth / pending_native_transport_requests >100 — paged 30+ seconds ahead of the trailing write_latency p99 >100ms. Mitigation: counter shard-splitting at the application layer — vote-svc writes a vote into one of N=16 sub-partitions chosen by hash(user_id), and the ranker sums on read. Cost: 16-way fan-out on the ranker's tally read = 16K Cassandra reads/sec for the #1 post at 1× scale, 640K at 10× — the ranker's 5s per-post tally cache (see ranker.keyChoices) is the load-bearing mitigation for this read-amp. (2) MV lag drift: monitor scrapes db_score - mv_score per-post; alert at >5% drift triggers outbox drain + re-rank. (3) LWT contention storm under the same hot-partition load: external retries on LWT multiply Paxos contention — the e13 edge uses retryPolicy=none for exactly this reason.