Instagram News Feed — a worked solution

What each component is for

Mobile

Issues GraphQL/HTTPS for feed-open and post-create. Maintains a client-side seen-set bloom filter so re-issuing a cursor after a network blip doesn't render duplicates.

Why it exists. Mobile is the product — web is a thin shadow. GraphQL persisted-query IDs keep raw queries off the wire (bandwidth + abuse-surface win); the opaque server-minted cursor keeps rank state authoritative. Worth the client-protocol discipline at 1.5B DAU.

When it fails. App crash mid-feed-open or socket drop loses the in-flight cursor — client retries with the persisted cursor; duplicates filtered by the local seen-set bloom. Detection: client telemetry on cursor_retry_rate; mitigation is the bloom plus server idempotent GET (rerunning the same cursor returns the same page within 15 min).

CDN · Edge POPsAkamai + Meta FNA

Terminates TLS at ~85 PoPs; serves signed media URLs from edge cache (30d immutable TTL); origin-shields the regional ingress for hot creators so a viral asset triggers ONE origin fetch shared across PoPs. Feed JSON passes through with Cache-Control: private, no-store — ranking is per-(user, ranker_version) and cannot be edge-cached.

Why it exists. Media is 95%+ of egress bytes; serving it from origin is the cost model breaker (~17 GB/s/region origin egress at peak). The Cornell SOSP study on Facebook photo caching reports 65.5% browser hit rate; the remaining 35% would otherwise hammer the backbone. We rejected origin-only serving — the math on backbone bandwidth alone justifies the edge tier ten times over.

When it fails. PoP failure → BGP withdraws the anycast prefix; clients route to next-nearest within ~3s, no client action. If origin shield misses on a viral asset, the origin fetch is shared across PoPs by request-coalescing — without it, a single PoP miss can fanout to thousands of upstream PoPs. Detection: edge origin_offload_rate per asset; alert if a single blob_id exceeds 100 origin GETs/min.

API Gateway · GraphQLEnvoy + GraphQL persisted queries

Resolves persisted-query IDs to internal RPC routes; enforces token-bucket rate limits per (user_id, surface) at 30 rps / device with burst 60; hedges ranker-bound calls at p95+10ms for tail-latency mitigation; carries the auth bearer to Auth/Identity for ACL enrichment before fanning out to services.

Why it exists. We need a tier that owns abuse + auth + rate-limit decisions outside the service mesh — services should never re-implement these. Considered putting rate limits at Envoy sidecars only, but per-user/per-surface limits need a fast distributed counter (Redis-backed) and that's gateway-level state, not sidecar-level. A single gateway tier also lets us A/B route surfaces at one chokepoint.

When it fails. Gateway tier saturates → rate-limit Redis becomes contended → tail latencies leak through. Detection: gateway 429_rate derivative + p99 latency on rate-limit lookup. Mitigation: shed read traffic first (post-create stays admitted), drop hedging budget to zero, fail-open on the rate limiter for 30s while the Redis pool catches up. Cert expiry: SPIFFE/SPIRE auto-rotation with 30/14/3-day alerts.

Auth / IdentityOIDC + recency-tokened ACL service

Validates the session bearer; resolves the user's effective ACL set (blocks, mutes, restricted accounts); attaches a recency_token = (region_lsn, mutation_ts) so downstream services can detect ACL mutations newer than what the regional follower has replicated. Rejects requests with stale recency tokens by routing them to the master region.

Why it exists. ACL mutations (block / unfollow) MUST be read-your-writes — eventual consistency here is a privacy SEV, not just stale UX. We rejected per-request leader reads (would 5x the master-region load); the recency-token pattern lets the common path read from the regional follower while only the post-mutation window pays the cross-region cost.

When it fails. Auth tier degraded → all surfaces return 401 (fail-closed). This is the right answer — better to serve nothing than serve someone else's feed. Detection: 401 rate vs baseline + auth p99. Mitigation: pin to last-known-good session (60s grace) for already-authenticated requests; drop new-session creation but keep refresh working. Master-region failover gap (the 30s MySQL RTO window): recency-tokened reads have NO master to route to. Behavior in that window is serve-stale-tagged-degraded: read from the highest-LSN regional follower and stamp the response degraded_acl=true; client surfaces a 'recently changed' banner on ACL-affected views. The privacy compromise is bounded to the 30s gap × the rate of ACL mutations within that window — defensible because the alternative (fail-closed for 30s on every recently-mutated session) is worse UX than a brief stale read with an explicit indicator. SLO: recency_token_master_route_rate and recency_token_degraded_serve_rate — both alert if non-zero outside known failover events.

Service · Feed ReadHack/HHVM (or modern equivalent: Rust + Tokio)

On feed-open: validate ACL recency, check the Cursor Cache for the requested page (continuation hits cached page; first page misses), call Candidate Generator → Ranker, hydrate post + author objects through the Object Hydration cache, emit cursor-with-next, return JSON. Owns the per-request deadline propagation: the 600ms feed-open p99 budget is sliced and pushed downstream as gRPC deadlines.

Why it exists. The feed surface is the fattest read in the product — it touches graph, posts, ranker, features, hydration. Splitting the orchestration into a dedicated service (vs putting it in the gateway) lets us scale independently (600 replicas at peak) and isolates the feed's blast radius from other surfaces that share the gateway. We rejected per-page Lambda-style serving — cold-starts at 350K QPS would be brutal.

When it fails. Adaptive concurrency starts shedding → users see chronological fallback feed (degraded flag in response). Worse: ranker loop with retries → cascade. Detection: degraded_response_rate per region + ranker call retries derivative. Mitigation: circuit-breaker on the ranker after 30s of degraded above 5%; the chronological fallback is the load-bearing safety net (ranker can be down for an hour and the product still works, just gets stale).

Service · Candidate Gen + ESRFollow-graph retrieval + lightweight gradient-boosted ESR

Retrieves candidates from two sources and merges: (1) follow-graph recent posts via TAO read of posts_by_author for friends, (2) precomputed denormalized feed-index for fanout-on-write recipients. Runs a lightweight Early-Stage Ranker (gradient-boosted, ~5ms/candidate) to drop from ~5000 retrieved to ~500 sent to the LSR.

Why it exists. Single-stage ranking 5000 candidates with the heavy MTML model would burn 15× the GPU. The retrieval → ESR → LSR funnel is the standard architecture that Pinterest, TikTok, and YouTube all use. We rejected pure pull-on-read (won't satisfy <600ms p99 once the candidate set exceeds 500) and pure fanout-on-write (the celebrity-followers explosion melts inbox-write throughput at >100K followers).

When it fails. Feed-index read stalls or a slow TAO shard → recall drops, feed degrades to 'most recent from your follow list only' (recall mode). Detection: candidate-set size derivative + recall-evaluation canary. Mitigation: fall back to TAO follow-graph posts only (the 'recall mode'); feed still works, just thinner. Hot creator partition: a viral post is in everyone's candidate pool, so the partition holding their posts_by_author list takes 10-50× median QPS — mitigated by promoting that key to in-process LRU at every Cand Gen replica.

Service · Ranker (LSR)MTML model on MTIA / GPU + ONNX runtime

Receives ~500 ESR-output candidates; fetches per-(user, candidate) features from the Online Feature Store; runs a multi-task multi-label model that predicts P(like), P(comment), P(share), P(dwell), P(skip); blends them via the current value-model weights into a final score; returns top-50 with calibrated scores. Per-call CPU budget: ~150ms wall (GPU inference ~30-50ms + feature fetch ~20ms + post-processing).

Why it exists. Ranking is the product. A chronological feed has 30% lower session length per Meta's own A/B disclosures; the entire candidate funnel exists to feed this model. We rejected putting the LSR inline with Cand Gen (different scaling profile — Cand Gen is CPU/RAM-bound on the ANN index, LSR is GPU-bound on the MTML model; merging would either over-provision Cand Gen or starve LSR).

When it fails. Model server GC pause or feature-fetch slowdown pushes p99 from 50ms to 600ms; Feed Read times out, retries amplify, queue depth blows up — classic ranker timeout cascade. Detection: ranker request_queue_depth derivative + retry_ratio>0.3 + downstream feature-store QPS spike with no organic traffic increase. Mitigation: adaptive concurrency at Feed Read clamps in-flight, retry budget capped at 10%, circuit-break to chronological fallback. The fallback is what stops the cascade; retries don't help — they hit the same slow model.

Service · Post WriteRust + Tokio, idempotent client_request_id key

Validates the post payload + media blob_id; HEADs the blob in Object Store · Media as a precondition to detect orphaned post-references (returns 422 if the upload didn't complete); writes the canonical post row PLUS an outbox event row to TAO/MySQL in a SINGLE SQL transaction (the load-bearing 2-participant outbox pattern); does NOT publish synchronously. Worker · Outbox Poller picks up the outbox row asynchronously and publishes post.created to the New-Post stream. Returns 201 with the canonical post_id once the SQL transaction commits.

Why it exists. Splitting writes into their own service tier keeps post-create out of the Feed Read fast path and lets us put creation-specific abuse rules (per-account daily limits, automated content checks, watermarking handoff) here without touching the read tier. We rejected gateway-side write handling (would mix abuse-policy responsibilities with edge concerns) and considered queueing all writes through Kafka first (durability win, but now post-create returns 202 'maybe' which the product team rejected — users want a hard 201/4xx).

When it fails. Primary shard fails mid-write → only in-flight (un-acked) writes are at risk; they return 503 and the client retries with the same client_request_id after failover (RTO 30s). Acknowledged commits are safe: semi-sync means an acked COMMIT is already on ≥1 same-region follower, so intra-region failover to the most-caught-up follower loses ~0 acked writes (RPO ≈ 0 — nonzero only in the degraded case where semi-sync timed out into async fallback). Detection: write success rate per shard + replication-lag-at-failover. The user-visible failure is 'post is publishing…' for the failover window — bounding acked-then-lost posts to ~zero is exactly why we run semi-sync rather than pure async. Orphaned blobs (client uploaded but never POSTed metadata) are GCed lazily by a Tectonic-side reference-counter sweeper at 7 days — accepted side effect of the presigned-upload flow.

Worker · Fanout (denorm)Kafka consumer group, Rust

Consumes the New-Post stream; reads the Followers table (hash(followee_id)) to enumerate a creator's followers on one shard; for each post by a creator with <100K followers (the cutoff), fans out into the recipient feed-index in TAO/MySQL — upserts (recipient_user_id, post_id, score_at_insert) rows keyed (recipient_user_id, post_id) into the recipient's feed-index partition (idempotent, so a re-published post.created from at-least-once outbox delivery is a no-op). Skips creators above the cutoff (those are pull-on-read at feed-open time). Maintains a 32-deep DLQ topic for poison messages.

Why it exists. Hybrid fanout is the canonical Twitter/Meta lesson: pure fanout-on-write melts on celebrities (615M follower posts = 615M writes, 3.4-hour tail at 50K writes/sec/shard); pure fanout-on-read makes feed-open a multi-shard scatter-gather. Cutoff at 100K is empirically tuned — above this the writeback cost beats read-amortization (see deepDive fanout-cutoff). We considered scaling all to fanout-on-read like Twitter pre-2014 — the resulting feed-open p99 doubled in a 2017 Twitter retrospective.

When it fails. Consumer group lag spikes (rebalance, slow downstream, partition skew) → new posts don't appear in followers' feeds for X minutes; a post's reach decays exponentially with time-since-post so a 10-min lag destroys most of a post's lifetime value. Detection: Kafka consumer_lag per partition + post_published_to_visible_in_feed p99 SLO. Mitigation: per-partition autoscaling; shadow chronological reads from posts-DB as fallback when lag > 60s; priority-lane topic for celebrity posts so they never queue behind the long-tail.

Worker · Engagement AggregatorFlink + RocksDB state

Consumes the Engagement stream of passive ranking signals (dwell, skip, impressions) emitted by Feed Read; updates the Online Feature Store with the derived features (dwell-time per (user, content_type), skip-rate, impression recency) at sub-second staleness.

Why it exists. Online features are the difference between 'this looks like an Instagram feed' and 'this looks like a 2014 Instagram feed' — recency of passive-behavior signals is what makes a feed feel alive. We rejected synchronous feature updates inline on the read path (would couple Feed Read latency to feature writes) and rejected purely batch features (the resulting 24h staleness ages out the ranking loop within hours).

When it fails. Aggregator lag → feature-store features are stale → ranker scores with 1-hour-old passive signals → recommendations look stale to users. Silent failure — no errors, just bad ranking. Detection: per-feature freshness_seconds p99 + ranker input distribution drift detector. Mitigation: ranker-side fallback to default values when a feature exceeds its staleness budget; dual-write online + nearline path so a single pipeline failure doesn't blank features.

Cache · Object HydrationMemcached + McRouter (TAO follower role)

Per-region read-serving cache for post objects, author profile objects, and ACL flags. Read pattern: get_multi([post_id, author_id, ...]) for the 50 hydration items per feed page; on miss, McRouter issues a lease token and the missing keys fall through to the Graph DB · TAO+MySQL replica. Concurrent misses on the same key wait on the lease holder rather than stampeding the DB — the canonical Meta NSDI '13 lease mechanism. The hydrated post object carries denormalized like_count / comment_count fields (materialized onto the post at write / fanout time), so Feed Read renders engagement counts straight from hydration without a separate counter read.

Why it exists. Without this layer, every feed-open does 50 round-trips to MySQL — at 350K feed-opens/sec peak, that's 17.5M MySQL reads/sec, ~10x what sharded MySQL can serve. The TAO paper reports 96.4% follower hit rate at >1B reads/sec; we sized for the same. We rejected client-side caching at Feed Read replicas only — the working set (~22.5 TB hot) doesn't fit in 600 service replica RAMs without sharding, and a shared shard-aware cache is exactly what memcache+mcrouter is.

When it fails. Cluster restart → cold cache → every read misses to MySQL simultaneously (canonical thundering herd) — without leases, MySQL gets crushed for 2-10 min until cache warms; with leases, concurrent missers wait on the first miss-holder so MySQL sees ~1 read per key, not N. Detection: lease_hold_time_p99 + miss-rate cliff + downstream MySQL QPS. Mitigation: gradual restart (1% pool at a time, never all at once); leases ARE the mitigation — they're load-bearing, not optional. Single-shard loss: McRouter routes the shard's keys to a backup pool (the lease-holding miss falls through to MySQL, then writes back to the backup); cache state is not durable, so 'failover' here means 'reroute requests' not 'promote a replica'. Cache poisoning of celebrity object: 10% bypass-cache canary on hydration to detect drift between cache and DB.

Cache · Ranked CursorRedis cluster (cluster-mode, 16384 slots)

Stores the ranked page for (user_id, session_id, ranker_version) so continuation pages don't re-rank — the ranker fires once per first-page; pages 2..N read from this cache. Each entry: an ordered list of ~50 post_ids + ranker scores + candidate_pool_snapshot_id + seen_set_bloom. Entry size ~2 KB; cardinality ~430M live entries at peak (50M concurrent sessions × 4.3 pages × 2 ranker versions live).

Why it exists. Without this, every continuation-page hits the ranker → 4.3x ranker QPS → 4.3x GPU spend. The continuation cache turns 350K feed-opens/sec into 81K rank/sec — a 4.3x cost reduction on the most expensive tier. We rejected putting cursors in client local storage (clients lie / can be replayed; rank state must be server-authoritative); rejected materializing in MySQL (~860 GB working set at 2 KB/entry needs in-memory).

When it fails. Cluster slot migration during a resharding event → cursor lookup fails for keys mid-migration → user sees 'page reset' to first-page. Detection: cursor_lookup_miss_rate derivative during shard-ops. Mitigation: page-1 fall-through always works (it just calls the ranker); the failure mode is degraded user experience (lost scroll position), not broken — acceptable. Ranker version bump: cursors carrying old ranker_version are honored for an overlap window of 1h, then return cursor_invalidated=true and client re-fetches page 1.

Graph DB · TAO + MySQLMySQL 8 (InnoDB) + TAO graph cache, ~256 logical shards

Canonical store for the social graph (follows / blocks / mutes) and the posts table (post_id, author_id, blob_id, created_at, content_metadata). Sharded by hash(user_id) for graph and hash(post_id) for posts; a feed-open hits the recipient's graph shard for the follow set and N post shards for hydration. Single primary per shard in the master region; semi-sync replication to two followers in the same region + async replication to follower regions.

Why it exists. We need transactional uniqueness on (user_id, follower_id) graph edges and on (post_id) — without it, double-follow or duplicate post_id corrupts the feed. Sharded MySQL is the canonical Meta+Twitter+Pinterest+Discord choice for this profile. We rejected DynamoDB (would work, but loses SQL escape-hatches when product asks for 'top 100 takedown candidates' / abuse queries), rejected Spanner (write latency 5x higher cross-region — feed-write SLO breaks).

When it fails. Primary shard failure mid-write → only in-flight un-acked writes at risk (acked commits are already on a semi-sync follower, RPO ≈ 0); clients retry with client_request_id after 30s RTO. Hot shard on viral creator (everyone's pulling their posts_by_author list at once) → mitigate by promoting hot keys to in-process LRU at every Cand Gen replica + read replica scaling. Cross-region replication lag spike → privacy SEV (block/unfollow takes 30s to land in EU); detection: tao_follower_leader_replication_lag_p99 > 5s; mitigation: recency-tokened reads route to master region during the lag window.

KV Store · Online Feature StoreManhattan-style RocksDB-backed KV (FBLearner Feature Store role)

Low-latency point lookup of features keyed by (user_id, feature_name) and (post_id, feature_name). Ranker fetches ~20 features per (user, candidate) pair × 500 candidates = 10K feature-fetches per rank call; budget 20ms total → 2µs per fetch (cluster mget). Updated by Worker · Engagement Aggregator with sub-second staleness for dwell/skip velocity features; offline graph features refreshed daily.

Why it exists. Ranker model serving requires features at inference time; features must be (a) extremely low latency (microseconds, not milliseconds — there are 10K per request), (b) consistent enough that the same (user, candidate) pair scores the same within a session. Storing features in MySQL would 100x feature-fetch latency; storing in memcached would lose the per-feature TTL granularity. Manhattan-style KV (RocksDB-backed, RF=3) is the canonical Twitter+Meta choice for this profile.

When it fails. Feature staleness spike → ranker scores with stale features → recommendation quality drops (silent failure, no errors). Detection: per-feature freshness SLO + ranker input distribution drift. Mitigation: ranker-side fallback to default feature values when stale beyond budget. Hot feature key (e.g. a celebrity's profile features) → mitigated by the standard Manhattan client-side cache + read replica scaling.

Object Store · MediaHaystack (hot) + f4 (warm) + Tectonic (cold)

Three-tier media storage. Hot (first hours/days, sub-1-read/day after that): Haystack — needle-in-haystack append-only volumes, RF=3 within DC, in-memory index, p99 read 5ms. Warm (>1 month, low read rate): f4 — Reed-Solomon(10,4) intra-DC + XOR cross-DC, effective replication factor 2.1× (saved 87 PB on a 65 PB corpus per the OSDI '14 paper). Cold + analytics: Tectonic — exabyte multitenant FS, the largest cluster is ~1,590 PB / 4,000 nodes / 10B files.

Why it exists. Storing all media in Haystack RF=3 forever costs 3.6× the raw byte count; at exabyte scale that's a procurement-budget catastrophe. The tiered model is what makes the unit economics work. We rejected pure S3 (third-party cost at this scale is unfavorable; cross-region semantics don't match Meta's regional-master pattern), rejected pure Tectonic (read latency too high for hot blobs).

When it fails. Haystack volume failure → blob unreadable until volume restore from RF=3 peers (RTO seconds via in-memory index re-replication). f4 single-DC loss → cross-DC XOR reconstructs (RTO minutes, no data loss). Tectonic cluster degraded → cold reads slow but never fail (RPO 0). The actual scary failure: a blob_id referenced from a post_id but the blob itself was lost mid-tier-migration. Detection: blob_404_rate_per_post_age_bucket (a 404 on a >7-day-old blob is a tier-migration loss; on a <1-day-old blob is a Haystack incident). Mitigation: clients receive a 404 from CDN, fall back to a 'media unavailable' placeholder; the post itself stays in feed (textual content preserved); a separate reconciliation job re-uploads from device-side cached copy if available. Reference counting is a precondition (prevents premature GC), not the recovery mechanism.

Stream · New-Post IngestKafka 3.x (or Scribe-equivalent)

Carries post.created events from Post Write to Worker · Fanout. One event per post (not per recipient). Partitioned by hash(author_user_id) so all posts by the same creator land on the same partition (consumer affinity for fanout-batching). Retention 72h (long enough for a consumer outage + replay).

Why it exists. The fanout-on-write denormalization can't run in the synchronous post-create path without blowing the post-create p99 budget — a 50K-follower post writing 50K rows takes 1-10s. Decoupling via Kafka turns post-create into a 200ms operation and fanout into a background pipeline. We rejected RabbitMQ (per-message ack overhead is wrong for this throughput profile) and SQS (regional, lock-in, retention too short for replay).

When it fails. Broker outage with min.insync.replicas=2 → the Outbox Poller's publish fails and retries; post-create is unaffected and still returns 201 (the post is already durable in MySQL — the publish is off the critical path by design, so a broker outage delays fanout, it cannot 503 a create). Detection: producer error rate + outbox_publish_lag_p99. Mitigation: the outbox poller drains from MySQL once the broker recovers; fanout is delayed but no post is lost. Consumer-side outage: Worker · Fanout falls behind → see Worker · Fanout failure mode.

Worker · Outbox PollerRust + Tokio cron-loop, 100ms tick

Polls the outbox_events table on each TAO/MySQL shard at 100ms intervals; SELECTs rows where published_at IS NULL ORDER BY id LIMIT 256; publishes each as a post.created event to the New-Post stream; UPDATEs published_at once Kafka acks. The single load-bearing component for 'posts MUST not be lost; events emit iff post commits' — without it, the outbox pattern is just an unused MySQL table.

Why it exists. We rejected having Post Write publish synchronously to Kafka (would put a Kafka-ack in the post-create critical path, blowing the 900ms p99 SLO when a broker has a transient issue) and rejected dual-writes (would lose the iff guarantee on Kafka-failure-after-MySQL-commit). The poller decouples durability from publish, at the cost of a separate component to operate.

When it fails. Poller dies for a shard → outbox table grows unbounded; new posts commit but events never publish → followers don't see them via the fanout pipeline (the post is still durable, just invisible until the poller catches up). Detection: outbox_publish_lag_p99 per shard (target <500ms); outbox_table_row_count per shard derivative. Mitigation: etcd lease auto-fails-over to a peer poller within 5s; the outbox table itself is the durability mechanism — even multi-hour poller outage is recoverable, just user-visibly stale.

Queue · Fanout DLQKafka topic, manual replay tool

Receives messages from Worker · Fanout that exhausted 32 retries with exponential backoff (poison messages — schema mismatch, downstream-permanently-broken, etc.). Holds 7-day retention so on-call has time to investigate; a manual dlq-replay ops tool re-publishes selected messages back to the New-Post stream after the underlying issue is fixed.

Why it exists. Without a DLQ, poison messages either block the partition (if stop-on-error) or get silently dropped (if skip-on-error). Both are bad: blocked partition = ALL posts on that partition can't fanout; silent drop = the affected posts never appear in followers' feeds with no signal. The DLQ trades 'delayed fanout for poison messages' for 'no other posts blocked + audit trail'. We rejected sending poison messages back to a 'parking' topic on the same cluster (capacity contagion) and rejected unbounded retry (worker-thread starvation).

When it fails. DLQ depth grows unbounded → indicates a sustained poison-message source (schema drift, codec-incompat) → on-call investigates. Detection: dlq_depth derivative per partition; alert on >100 messages in 1h. Mitigation: replay tool after fix; in extreme cases (schema break) the DLQ messages are abandoned with an audit trail.

Stream · Engagement EventsKafka 3.x

Carries passive ranking signals (dwell, skip, impressions) emitted by Feed Read to Worker · Engagement Aggregator. Partitioned by hash(post_id) so all signals on one post land on one partition (consumer can locally aggregate without cross-partition reads). Retention 24h.

Why it exists. Passive signals feed the online-feature update path. Without the stream, the aggregator would tail Feed Read's request path directly → fragile, fan-in-on-service. The stream is the event-sourced spine for ranking signals — it lets new downstream consumers come online without coordinating with Feed Read.

When it fails. Consumer lag → online features go stale → ranker scores with old features → recommendations look stale (silent failure). Detection: per-partition consumer lag + per-feature freshness SLO at the feature store. Mitigation: dual-write online + nearline path so a single-pipeline failure doesn't blank features; ranker-side default-value fallback when features exceed staleness budget.