Like Button at Scale — a worked solution

What each component is for

Client (SDK)iOS / Android / Web SDK

Renders the post with its like count and hearted/un-hearted state, captures taps, sends POST /v1/likes with an Idempotency-Key, and applies an optimistic UI flip on tap so the user sees their own write before the server has even ack'd. Reconciles the local state on the next GET.

Why it exists. The SDK owns read-your-write for the liker. The backend is eventually consistent across viewers; the liker MUST see their heart flip immediately even if the cache / CDN / replica has not yet caught up. Client-side optimism + a server-confirm token is the only honest way to deliver that without an active session pin into every replica.

When it fails. Network drops mid-write: client retries the same Idempotency-Key — server dedups, no double-count. Optimistic state stuck if server never acks → SDK reconciles on next GET (state from server wins). A bug in the optimistic state machine = thousand-tap users seeing wrong heart state → caught by the count_flicker_per_user metric backend-side.

CDN (edge cache)Cloudflare / Fastly

Caches the per-post {count, topReactions} JSON at the edge for 5–30 s (TTL is per-post heat — viral posts get 5 s, cold posts 30 s; heat signal comes from the gateway's hot-key sampler). Personalized hasLiked is bypassed — POSTs and any request bearing an Authorization header are pass-through to origin.

Why it exists. The dominant read shape is many viewers reading the same hot post's count. Without an edge cache, ~99% of cold-load reads hit origin and a viral post's 5M+ reads/s would saturate every read-svc pod globally. The CDN absorbs 80–95% of read QPS; origin is sized for the residual 5–20% plus the hasLiked traffic.

When it fails. Regional POP outage → traffic fails over to neighbor POP, colder cache, miss rate spikes from ~5% to ~20%, origin sees 4× more traffic; count-cache + read-svc must absorb. CDN serves stale count after post-delete / mass-unlike → purge-fanout failure is the primary cause; safety net is the 30 s natural TTL.

API GatewayEnvoy + WAF + rate-limit + hot-key sampler

Terminates TLS, applies WAF, enforces per-user (100 writes/min) and per-IP (1000 writes/min) rate limits on POST /v1/likes, mTLS-binds to downstream services, samples 1% of traffic for hot-key detection via a count-min sketch, and routes by URL: POST /likes → write-svc, GET /posts/:id/likes → read-svc.

Why it exists. A single common edge that does the things every service would otherwise re-implement: TLS, authn, rate-limit, request tracing. Pulling it out lets read-svc / write-svc focus on the like-specific work. The hot-key sampler lives here because the gateway sees the un-salted, un-sharded request URLs — downstream services see post_ids already smeared by sub-counter shards and can't tell which key is hot.

When it fails. Pod crash: LB ejects within 5 s; cluster of 40 absorbs one loss with no user impact. Hot-key sampler misses a flash spike (sub-30 s viral curve): count-cache shard saturates for 30–60 s before key-split kicks in — mitigated by always-replicating count:* to 4 hot pools even before the sampler flags. WAF false-positive blocks a legit author: page on the waf_block_rate_per_authed_user SLO.

Like-Read ServiceGo / gRPC

Stateless service that assembles the per-post like response: GETs the aggregate count:{post_id} from count-cache (or SUMs sub-counters / falls through to counter-store on miss), and for authenticated callers checks hasLiked(user, post) from a per-user Bloom filter, falling through to a per-(user, post) lookup in edge-store. Returns the merged payload in a single round-trip.

Why it exists. Splitting the read and write services lets each scale independently — reads run 100–500× writes (~18M/s peak vs ~93K/s peak, per the capacity anchors) and need wildly different connection-pool sizing, GC tuning, and JIT warmup than the write side. Co-locating them would mean a write deploy could ship a read regression on the busiest endpoint in the product.

When it fails. count-cache shard unavailable → fall through to counter-store; p99 jumps from 5 ms to 100 ms — the explicit graceful-degradation budget. counter-store unavailable too → serve {count: stale, lastFresh: ts} from the last good response (1 m TTL); UI degrades to 'about N' rounded. The read NEVER 5xx's — likes are decorative, not critical to the product loading.

Like-Write ServiceGo / gRPC

Accepts POST /v1/likes with an Idempotency-Key. GETs the key from idem-store to dedup retries; if new, issues a Cassandra BATCH against the edge-store partition for the user that writes BOTH the edge row AND a row into the outbox table (partition-local atomicity — both succeed or both fail). After the batch acks, publishes to Kafka likes.committed and SETs the idem key with {state, intent_uuid, event_id}. Returns 202 once the publish acks. If the publish fails after the batch commit, the outbox row remains and a relay process drains it on the next pass — the like is durable and the event WILL appear, late.

Why it exists. Outbox separates 'durable enough for the user's 202' from 'fanned-out to all downstream'. Edge-store is the system-of-record for (user, post, state) — a viewer's hasLiked query reads from it. Kafka is the spine for everything else (counters, anti-abuse) — without a single event stream, each consumer polling edge-store would melt it.

When it fails. Edge-store quorum loss → write fails fast with 503 + Retry-After (better than hanging). Kafka publish times out after edge-store BATCH commit → outbox row survives; relay drains within 5 s; consumer dedups via intent_uuid (NOT event_id — two write-svc pods retrying the same intent generate DIFFERENT event_ids but the same intent_uuid, and the aggregator dedups on intent_uuid). Idem-store down → fail-CLOSED (refuse the write); accepting an un-dedupable write risks double-count, the worse outcome.

Idempotency StoreRedis Cluster (24h TTL, AOF-persisted)

Key idem:{user_id}:{post_id}:{intent_uuid} → {state, ts, event_id}. write-svc GETs before any side-effect to dedup; SETs the key after Kafka publish + edge-store commit, with TTL=86400 s.

Why it exists. At-least-once retries are the default on every layer — mobile SDK retries on connection drop, the gateway retries on 503, internal RPC retries on transient errors. Without a dedicated dedup store, every retried POST becomes a double-increment. The dedup window has to outlive the worst-case retry pipeline (mobile app restored from background after a flight = measured 18 h tail), so 24 h is the floor.

When it fails. Shard unavailable → fail-CLOSED on writes that route to it (refuse rather than risk double-count); read-svc unaffected. Cluster restart → 5 s of writes may 503; mobile SDK retries succeed once cluster is back; no data loss because the post-commit SET hasn't happened yet so the retry simply re-runs the full path.

Counter CacheRedis Cluster (mcrouter-style hot-key replication)

Stores the per-post aggregate count:{post_id} and the per-shard sub-counter set count:{post_id}:s0..s63. read-svc GETs the aggregate (5 s TTL); on miss it SUMs the 64 sub-counters in one MGET. Aggregator updates sub-counters on each Flink emit and re-SETs the aggregate with a monotonic generation_id (slow emitter's stale write LOSES the SET race).

Why it exists. A single Redis INCR on count:{post} for a viral post is a hot-key disaster — one shard saturates at ~100K ops/s while the rest sit idle. Per-shard sub-counters spread the write across N keys; the read pays N GETs but those are 1–2 µs each. The pre-aggregated count:{post} lets ≥95% of read-svc reads do a single GET; only on TTL expiry / cache miss does the SUM run.

When it fails. Shard primary failure: 15 s failover RTO (cluster mode, async repl), 5 s of recent INCRs may be lost — recovered by Flink replaying the last 5 s window from Kafka. Cluster restart cold: all SUM operations hit counter-store; counter-store sized to absorb 10× steady miss rate for ~5 min. Hot key on an under-replicated post: one read pool saturates → mitigated by hot-key detector → key-split bumps replication factor.

Like-Edge StoreCassandra / ScyllaDB (RF=3, LOCAL_QUORUM)

Source-of-truth for (user_id, post_id, state, ts, reaction_type, void_flag) rows. Sharded by user_id (so hasLiked queries are single-partition). Write path UPSERTs with IF state != ? semantics; read path is point-lookup by (user_id, post_id) for hasLiked.

Why it exists. We need an authoritative source for who liked what, separately from the counter. Counters can be approximate; the like edge cannot (a user must see whether they liked a given post; we need to support 'undo' and GDPR audit). Sharding by user_id — not post_id — optimizes the hot read pattern: hasLiked-for-this-feed-of-50-posts becomes 50 same-shard lookups, not 50 cross-shard scatter-gathers.

When it fails. Single AZ loss: writes continue with W=2 from 2 surviving replicas; no user-visible impact. Quorum loss (region-level partition isolates 2 of 3 replicas): writes BREAK in that region — write-svc fails fast → SDK retries → eventual consistency once partition heals.

Counter Store (CRDT)Cassandra/Scylla counter columns (PN-Counter per region × shard)

Stores per-(post_id, region, shard) counter cells. Each region writes only its own cells; the per-post aggregate is SUM(cell for cell in (post_id, *, *)). Aggregator's batched increments land here; read-svc SUMs on count-cache miss. Decrements from anti-abuse voids land here too (PN-Counter semantics).

Why it exists. Multi-region active-active writes on a single counter row would either fight (CAS retry storm) or lose increments (last-write-wins). A G-Counter / PN-Counter (per-actor partition + sum on read) is the only safe pattern — each region owns its slot, merging is commutative + associative, and a 5 min cross-region replication blip just means the global count is 5 min stale, never wrong.

When it fails. Single Cassandra node failure: writes continue at W=2; reads downgrade to LOCAL_ONE. Cross-region link severed: each region's count diverges by local_delta; on heal, CRDT merge converges automatically — no manual reconciliation. Hot post: per-shard cells balance the write load; cell-level hotspot mitigated by post_id key-split (post_id:0..post_id:7 sub-IDs at the aggregator).

Likes Event SpineKafka (256 partitions, RF=3, acks=all)

Single durable event log. Topic likes.committed (every committed like) and likes.voided (anti-abuse retractions). Aggregator and abuse both consume from likes.committed independently, at their own pace.

Why it exists. Outbox + Kafka is the only way to fan one write out to N consumers without N×point-to-point coupling. If the aggregator and abuse scorer each polled edge-store, the edge-store reads would explode and a deploy of any consumer could break the others. Kafka decouples — durability is in the log; consumers can lag, replay, or fail without affecting write-svc latency.

When it fails. Single broker loss: ISR shrinks, controller re-elects partition leaders; ~5 s of producer blocking, no data loss. Cross-AZ partition: producers retry against surviving brokers; one-replica writes refused (good — don't sacrifice durability for tail latency). Consumer lag spike on likes.committed: counts go stale (visible after ~10 s), abuse voids land late — surface as consumer_lag_seconds SLO alert.

Counter AggregatorApache Flink (KeyBy post_id, 1s tumbling window)

Consumes likes.committed AND likes.voided from Kafka, KeyBy (post_id, region), maintains a 1 s tumbling window in RocksDB state, and at each window-tick: (a) batches the deltas (positive for commits, negative for voids) into Cassandra counter-store updates, (b) writes the new aggregate to count-cache with versioned SET + issues a CDN Surrogate-Key purge. Dedups upstream via intent_uuid (NOT event_id) so two write-svc pod retries for the same user intent count once.

Why it exists. 1M individual INCRs into Cassandra / Redis would saturate both. Windowing collapses N+1 increments into one batched +N — at peak, a viral post sees 50K events/s collapsed into one +50K every second. Flink (vs custom code) for the at-least-once + checkpointed exactly-once-effect semantics and the operational tooling (Savepoints, RocksDB state, k8s operator).

When it fails. Flink job crash: checkpoint restore is ~2 min from S3; counts go stale during recovery — UI shows last-fresh count + a 'live' badge that hides. Slot OOM on a viral post: producer-side sub-keying spreads load across 8 slots. RocksDB compaction stall: Savepoint + restart on new pod; window state is bounded (1 s) so no replay storm. Poison event: 3 retries then publish to likes.dead-letter with the original event_id; per-partition unblocked.

Anti-Abuse ScorerPython + gRPC ML sidecar, consumes likes.committed

Consumes likes.committed, scores each like with an ML model (per-user rate, IP-cluster entropy, device fingerprint, account age — a calibrated probability that the like is bot-driven). For likes above the threshold, publishes a compensating event to likes.voided; the aggregator decrements; the edge-store row is marked void=true.

Why it exists. Synchronous anti-abuse in the write path would push the write p99 above the budget (model inference is 20–50 ms). Async-after-the-fact lets the like commit immediately (good for the user) and retroactively void the bot ones (the counter briefly shows a higher count, then settles). This is the trade-off accepted by every major platform: visible counts are eventually-consistent with the abuse-cleaned ground truth.

When it fails. Model service down: defer scoring (consumer buffers in Redis with 24 h TTL); when model recovers, drain and void asynchronously — at most 24 h of bot likes leak through. Pod crash mid-publish: void event NOT lost (offset not yet committed); restart re-publishes; aggregator dedups. False-positive cascade (bad model deploy): manual rollback + cleanup via re-publish of likes.unvoided (we keep that capability for exactly this case). Storm of bot likes: rate-limiter degrades the user's account (1/min) and pages the abuse team.

Config Storeetcd (Raft, 5-node cluster)

Distributes runtime control state: the gateway's count-min-sketch top-K hot post_ids (so count-cache and aggregator know which keys to sub-key / replicate), the per-post CDN heat hint (so cdn can drop its TTL on viral posts), and a small set of feature flags (canary %, kill-switches).

Why it exists. The hot-key story collapses without a control plane the dataplane can subscribe to. Pushing hot-key flags via Kafka would be slow (consumer lag = stale replication policy); pushing via direct RPC would couple gw to count-cache topology. etcd's Raft + watch semantics give us sub-second propagation with a sane consensus story for a small state size (<1 MB).

When it fails. etcd quorum loss: existing dataplane caches keep serving the last good config for 5 min, then expire — gradual degradation, not cliff. Stale hot-key flag: count-cache replicates a key that's no longer hot — wastes memory but doesn't break. Bad config push: a feature-flag panic mode is the same lever the SRE pulls.

Schema RegistryConfluent Schema Registry (managed)

Stores versioned Avro/Protobuf schemas for every Kafka topic. write-svc validates outgoing events against the registered schema before publish; every consumer (aggregator, abuse) fetches the schema on first contact with a new schema_id and caches it.

Why it exists. Without a registry, the failure mode is: producer adds a new reaction field, a consumer does not recompile, that consumer silently drops or crashes — and you find out via the drift reconciler 24 h later. The registry's FORWARD compatibility mode enforces 'events written with the new schema must remain readable by consumers still on the old schema' at PUBLISH time, so the bad write is rejected at the source rather than 24 h later. (BACKWARD is the opposite guarantee — new consumers reading old data — and would not protect the stale consumer.)

When it fails. Registry 503 on publish: write-svc fails the publish — rather than ship an un-validated event we 503 the user; mobile SDK retries succeed once registry's back. Bad schema change (incompatible): registry rejects at the producer — the bad deploy doesn't even ship. Cache poisoning: consumer holds a bad schema for 60 min; fix is forced cache-flush via config-store kill-switch.

Dead-Letter TopicKafka topic likes.dead-letter (32 partitions, 30d retention)

Receives events that aggregator / abuse couldn't process after N retries (typically: schema-incompatible payload, malformed UUID, downstream poison). Owned by the like-platform team; daily report of DLQ size goes to the team Slack. Runbook: inspect, fix root cause, optionally replay back into the source topic.

Why it exists. Without a DLQ, one poison message per partition stalls the entire partition forever — at 256 partitions of likes.committed, one bad event can stall 1/256th of all downstream processing indefinitely. A DLQ converts that into a quarantine the on-call can deal with on Monday morning rather than at 3am.

When it fails. DLQ topic itself unavailable: consumers fall back to logging the bad event + advancing offset (last-resort, alerted). DLQ grows unbounded: page when daily delta > 1000 events — usually means a producer / consumer version mismatch shipped without registry catching it.

OTel CollectorOpenTelemetry Collector (gateway sampling) → Tempo

Receives OTLP spans from every service (gw, read-svc, write-svc, aggregator, abuse). Applies tail-based sampling (keep 100% of error traces + p99-latency outliers + 1% of healthy), buffers locally on disk for 30 s of upstream-down resilience, and ships to Tempo for query.

Why it exists. A 'like' touches ≥4 services on the sync path and ≥3 more on async; without distributed tracing, debugging a 'why was that 99.9th-percentile write 800 ms?' takes hours of log-grep. Tail-sampling at the collector keeps storage cost bounded while still capturing every error and outlier — which is what on-call actually opens during a page.

When it fails. Collector pod loss: spans buffer locally on app pods for ~30 s; longer outage → spans dropped (alerted on otlp_export_failures). Tempo backend down: collector buffers up to disk limit; sustained outage = we're blind to traces but the product keeps working. Bad sampling config: drop too much = blind; keep too much = storage cost explodes — guard rail on traces_kept_per_second.

Metrics + AlertingPrometheus + Mimir (long-term) + Alertmanager

Scrapes every service for RED metrics (Rate, Errors, Duration) and every datastore for USE metrics (Utilization, Saturation, Errors). Mimir backs the historical store; Alertmanager evaluates SLO burn-rate rules and pages on-call via PagerDuty / Opsgenie.

Why it exists. Every SLI named in the observability section (redis_shard_qps_skew, consumer_lag_seconds, replication_lag_seconds, canary_drift_ratio, count_flicker_per_user, mm2_lag_seconds, traces_kept_per_second) is a metric that needs scraping, storing, and alerting. Without a metrics plane named in the diagram, those SLIs are imaginary.

When it fails. Prometheus pod loss: 15 s scrape gap; auto-replaced by k8s; Mimir backfills on next scrape — minimal data loss. Mimir storage corruption: short-term metrics survive in Prom local TSDB; long-term query degrades; recovery via S3 backup. Alertmanager misconfiguration: silent SLO breach — dead-man's-switch alert every 5 min as the canary.