Uber / Lyft — Match Drivers and Riders — a worked solution

What each component is for

Rider Client

Issues GET /eta as the user drags the pin (browse) and POST /requests on tap-to-confirm. Holds a persistent WebSocket through the gateway for live offer status (searching → matched → driver_arriving). Re-uses a client-generated request_nonce on every retry so the matcher dedupes.

Why it exists. We considered a stateless polling client (poll /requests/:id every 500 ms) — at 5K peak rider QPS x 4 polls/s that adds 20K spurious QPS to the gateway with no UX benefit. WS is the load-bearing choice; polling is the fallback when WS keepalive fails twice.

When it fails. App backgrounded mid-search drops the WS; searching state goes silent. Detection: client telemetry on ws_reconnect_rate and request_to_match_seconds p99. Mitigation: server-side reaper expires the request after offerTtlSec x 4; reconnect carries the same request_nonce so the matcher returns the cached in-flight result instead of a duplicate dispatch.

Driver Mobile

Streams GPS samples over WebSocket every 4 s while idle, every 1–2 s with a passenger or above 50 km/h. Receives offer envelopes via push (APNs/FCM) AND the same WS; POSTs accept/decline within the offer TTL. Snaps locations to the road graph client-side before send, so the server doesn't see oscillation around a parking lot.

Why it exists. Driver app is a bidirectional system citizen — both the heaviest write source (one heartbeat per driver every 4 s) and the receiver of the dispatch fanout. We rejected a polling-only design: at 1.5 M concurrent drivers, the reconnect storm on every gateway deploy would be ~375 K new TLS handshakes, which crushes the LB. Persistent WS with graceful drain on the server side absorbs it.

When it fails. GPS drift puts the driver in the wrong H3 cell — rider gets a driver-across-the-river match. Detection: server-side driver_jump_distance_m_per_s > 60 (> 220 km/h) is treated as suspect and the location update is dropped. Mitigation: Kalman smoothing + map-matching at ingest (locingest); legitimate jumps (tunnel, ferry) tolerate one cycle of stale before reverting. Driver-tunnel WS-drop while holding an OFFERED offer: WS-keepalive miss for >2 s triggers a fast-EXPIRE on the offer (DISCO transitions OFFERED→EXPIRED, riderapi rolls forward to next candidate).

Load Balancer · Global EdgeAnycast L4 + GeoDNS pin per city_id

Terminates the rider's TLS at the closest PoP, performs a city_id lookup against a small in-memory pin table (maintained out-of-band by ops; ~10K cities globally), and proxies to the gateway in the city's home region. Holds the global view 'NYC always lands in us-east-1 dispatch, SF always in us-west-2'. WS upgrades are stickied to the same region for the connection lifetime.

Why it exists. Without an explicit region-pin layer, our 'active-active per-city' claim is aspirational — a rider whose DNS resolves to us-east-1 but whose city is pinned to us-west-2 would dispatch against an empty driver index. The alternative (let each region's gateway forward across regions) doubles cross-region traffic on every request and adds 30–80 ms median; that's the entire ETA budget. Pinning at the edge is the only place this is cheap.

When it fails. GeoDNS resolves a rider in a city whose pin was just rotated mid-deploy → rider's request lands in the old region → returns 'no drivers near you' from the stale geoidx. Detection: per-city wrong_region_request_rate > 0 (the gateway rejects with a hint header). Mitigation: gateway returns 307 Temporary Redirect with the correct region's anycast endpoint; rider auto-retries; rotation procedure includes a 60s overlap window where both regions accept reads.

API Gateway · Public EdgeEnvoy 1.28 + WAF + JWT verifier

Inside one region. Validates the rider/driver JWT, enforces per-(IP, user_id, city_id, h3_r7_cell) rate limits, upgrades to WebSocket for the heartbeat and offer streams, and routes by URI: POST /requests → riderapi, GET /eta → riderapi, POST /offers/:id/accept → riderapi, WS /v1/driver/heartbeat → locingest. POST /requests uses ring-hash LB on the Idempotency-Key header so a retried request lands on the riderapi replica already holding its nonce future; other routes round-robin. Envoy outlier-ejection sheds slow upstreams without killing the whole pool.

Why it exists. Lyft's Envoy mesh exists exactly for this — uniform retry budgets, circuit breakers, mTLS upstream. We considered terminating WS at riderapi/locingest directly; that loses one place to enforce city-level admission control, which is the single most useful lever during a hot-cell event. The gateway is where 'shed before it cascades' happens.

When it fails. LB roll without drain: 1.5M drivers reconnect in a 5 s window, TLS handshake p99 spikes past 2 s, ingest brownouts city-wide. Detection: driver_ws_reconnect_rate > 10x baseline AND tls_handshake_p99 > 2s for 60 s. Mitigation: graceful drain + server-issued Retry-After 0–30 s uniform; pre-warmed Envoy capacity at +20% headroom over peak; emergency runbook flips the 5% rolling cap down to 1% during recovery.

Service · Ride RequestGo 1.22, gRPC, idempotency-key-aware

Owns the rider-side request lifecycle. On POST /requests: validate, fan out to ETA service for the offer envelope, then call DISCO with (rider_id, request_id, lat, lng, surge_version, eta_envelope). On POST /offers/:id/accept from a driver: validate offer is still in OFFERED state, route to DISCO for the CAS-commit. Persists the request row (SEARCHING → MATCHED | NO_DRIVER | CANCELLED) to Cassandra so poll/cancel resolve from any replica. Stateless except for an in-memory LRU of request_nonce → in_flight_promise so a retried request joins the same future — the gateway ring-hashes POST /requests on Idempotency-Key, so retries land on this same replica.

Why it exists. We considered making DISCO public (no orchestrator) — DISCO's match logic must not block on ETA composition or external Maps calls; coupling them turns one Maps blip into dispatch downtime. The orchestrator is where ETA + match are fanned out concurrently, the offer envelope is built, and idempotency is enforced before DISCO ever sees a duplicate.

When it fails. ETA service degrades; riderapi blocks waiting on the offer envelope; rider retries; pile-on at riderapi. Detection: riderapi_eta_call_p99 > 400 ms AND inflight_concurrent > 80% of limit. Mitigation: ETA timeout is 500 ms hard with circuit-breaker (no retry doubles the budget); on open, riderapi proceeds with surge=1.0 — degraded UX (price might shift on confirm) is better than 'searching forever'.

Service · ETA & PricingGo + Python (DeepETA model server)

Two endpoints: GET /eta (browse) and GET /offer-envelope (called by riderapi). Reads the per-cell surge multiplier from the surge cache, asks the Maps service for route distance/duration, applies the DeepETA residual model (a few-ms encoder-decoder on top of the routing ETA), and returns {eta_seconds, base_price, surge_multiplier, surge_version, ttl_seconds}. Stateless — model weights pulled on startup, refreshed via canary rollout.

Why it exists. Maps and surge can't be on DISCO's critical path: they are the long-tail latency sources. ETA is a separate process with its own SLO (200 ms p99) so a slow Maps call cannot push the matcher past its 600 ms budget. We considered baking ETA into riderapi; that fuses the fast-fail circuit breaker with the matcher's, which means a Maps outage takes down dispatch — unacceptable.

When it fails. DeepETA model rolls bad; ETA p99 jumps to 600 ms; riderapi hits its 500 ms timeout on every call. Detection: eta_response_p99 > 250 ms for 90 s + canary error budget. Mitigation: progressive rollout 1% → 10% → 100% per region with auto-rollback on p99 regression > 50 ms; fallback path returns routing ETA without the residual.

Service · DISCO DispatcherGo 1.22; in-memory cell index per shard; offer state machine

Receives match requests for a given H3 r9 cell, expands kRing(k=1..3, adaptive), pulls candidate driver_ids from the in-memory cell index plus a fast geoidx fallback, scores by ETA × acceptance-rate × fairness penalty, picks the best, CAS-claims the driver in geoidx, INSERTs the offer into rides Cassandra, fires push to APNs/FCM, and emits a transactional outbox event. State machine: OFFERED → ACCEPTED | DECLINED | EXPIRED, plus a fast-EXPIRE transition when the driver's WS goes silent for >2 s while holding an offer (avoids the rider waiting the full TTL).

Why it exists. This is the core of the product — the answer to 'who picks up this rider'. Sharded by RingPop on H3 r8 supercell (covers ~10 r9 cells, ~1 km^2) so a contiguous patch of map collapses onto one process. Shard-local in-memory state means kRing iteration is microseconds, not network calls. Alternative: every match call hits a central matcher reading from Redis on every kRing — the 4 RTTs per request blow the 600 ms budget at 5K req/s.

When it fails. RingPop ring instability mid-deploy: two owners both think they own cell C, both dispatch to driver D, driver gets two offers. Detection: offers_routed_to_wrong_owner > 0 (the rides CAS counter). Mitigation: lease-based ownership with monotonic epoch; rides reject stale-epoch writes — the second offer fails closed, riderapi falls back to the next candidate; ring re-converges within 10 s gossip cycles. Driver-WS-disconnect mid-OFFER: WS keepalive miss → fast-EXPIRE → next candidate within 2 s instead of 15 s wait.

Service · Location IngestGo; pipelined Redis client; Kafka producer; map-match filter

Terminates the driver WS heartbeat stream, runs a server-side map-match + Kalman filter on each sample (rejects > 60 m/s jumps as suspect), pipelines a SET driver:{id} + ZADD cell:{h3_r9} pair into geoidx (key=driver_id, value=(h3_cell, lat, lng, ts, status)), and publishes the same sample to the Kafka marketplace bus for surge + analytics consumers. No request fanout, no synchronous fan-in — fastest possible path from socket to KV.

Why it exists. Ingest is the write-dominant tier (1:125 read:write at the geo-index) and absolutely cannot share a process with the matcher — a slow matcher would back-pressure the heartbeat path and dispatch goes blind. Separate process means we can size locingest for write throughput (no big TLS handshake budget, fat batched Kafka producer) while DISCO sizes for matcher CPU.

When it fails. Map-match Kalman filter is too aggressive after a model bump: legitimate tunnel exits get dropped, cars vanish from the index for 30 s, riders in that area get 'no cars available'. Detection: dropped_samples_rate > 2% city-wide AND geoidx_active_drivers step-down. Mitigation: filter is feature-flagged per region; rollback on the regression alert; heartbeat samples are still emitted to Kafka raw so we can replay against a fixed filter.

Coordinator · RingPopRingPop (SWIM gossip + FarmHash CH ring), in-process

Maintains the consistent-hash ring of DISCO members and the H3-supercell → owner mapping. Every DISCO process embeds a RingPop client; gossip-converged membership lets any process forward a request to the current owner of a given cell. Holds the per-shard monotonic epoch that the dispatcher stamps onto every offer. The 'edge' from disco to ring is in-process (microseconds) — modeled separately on the diagram for clarity, not because there's a network hop.

Why it exists. Without an embedded ring, DISCO would need a centralized router (etcd / ZK on every match call) — that's a coordination hop on the 600 ms critical path, plus a second SPOF. RingPop's SWIM gossip is eventually consistent membership without a coordinator, paying the cost in bounded inconsistency during deploys (which we contain with the epoch). We considered Raft for membership; convergence latency at a 200-node ring is tighter than gossip, but Raft adds a leader and Raft elections during partial partitions are precisely the failure mode that paged on-call last incident — gossip fails open, Raft fails closed.

When it fails. Gossip storm during a 200-node deploy: membership views diverge for 20 s, cells oscillate ownership, in-flight offers route to the wrong owner. Detection: ringpop_membership_changes_rate > 5/min for 60 s + offers_routed_to_wrong_owner > 0. Mitigation: rolling deploy capped at 5% concurrency; epoch on every offer means the second 'owner' commits with a stale epoch and the rides CAS rejects it; double-dispatch is averted, the rider falls to the next candidate.

KV Store · Driver Live IndexRedis 7 cluster mode, 64 shards, 3-AZ, 2 read replicas/shard

Two key spaces: (1) driver:{id} → (h3_r9, lat, lng, status, ts, epoch) for the live driver record, TTL 30 s; (2) cell:{h3_r9} → SortedSet of driver_id scored by ts, refreshed on every heartbeat. DISCO does ZRANGEBYSCORE cell:{c} now-30s now × kRing(k) for candidate retrieval (allowed to read a follower replica, eventual), then a scripted CAS (WATCH driver:{id}; status==available; SET status=offered, epoch={E}) for the claim — the CAS routes to the leader for that shard, linearizable per record.

Why it exists. Hot working set is ~96 MB total (1.5M drivers × 64 B). A B-tree DB on disk would crush the cache hit rate at 1.875M writes/s peak. We considered Cassandra here — Schemaless is what Uber ran originally — but heartbeat write amp and the every-4-s churn turn the SSTable compaction cost into a CPU sink that costs more than just keeping it in RAM. Cassandra survives as the durable canonical (rides); Redis is the hot path.

When it fails. Async-replicated leader failover loses up to 5 s of recent heartbeats; followers promote with a stale view. Detection: cluster-mode +failover-end event paired with geoidx_lag_seconds > 5 (cluster-node-timeout default ~15 s). Mitigation: heartbeats refill the index in <2 cycles (8 s); during the gap, kRing reads degrade to LOCAL_ONE consistency, and the offer commit's CAS-on-driver-status fails closed when a driver record is missing — the matcher falls back to the next candidate rather than dispatching to a ghost. Hot-cell ZADD storm (concert end, ~2K ZADDs/s onto one shard): the cell sortedset is replicated across 2 followers so the kRing-read load shifts to followers, leaving the leader free for ZADDs.

Cache · Surge MultipliersRedis 7 cluster, 16 shards

Stores surge:{h3_r7} → (multiplier, version, computed_at) for every active res-7 supercell (~5 km^2, ~6M keys cardinality, hot subset ~500K). Read on every ETA call (~16K QPS peak — the 10:1 browse fan-out is already baked into that) — single GET per offer. Written by the surge aggregator at the end of each sliding window.

Why it exists. Surge has to be computable in milliseconds on every request — the surge multiplier is part of the offer envelope quoted to the rider. We rejected reading directly from Pinot (the analytics tier) on every request: Pinot p99 is 50 ms, single-digit ms is what the surge slot in the dispatch budget allows. The cache is the load-bearing decoupler between the windowed-aggregator latency (minutes) and the dispatch latency (milliseconds).

When it fails. Cache cold after a region-wide Redis restart: every ETA call falls back to surge=1.0 default for ~30 s while the aggregator backfills. Detection: surge_cache_hit_rate < 95% for 60 s. Mitigation: ETA service is allowed to serve surge=1.0 with a stale_surge=true flag rather than block on a slow refill; aggregator has a bootstrap-from-Kafka startup mode that reads the last 5 windows in parallel, restoring the hot subset of ~500K keys in under 30 s.

NoSQL DB · Rides & OffersCassandra 4.1, RF=3, QUORUM, 256 vnodes/host

Two tables: offer(offer_id PK, ride_id, driver_id, rider_id, state, surge_version, epoch, expires_at, ...) and ride(ride_id PK, driver_id, rider_id, status, pickup, drop, fare, started_at, ...). Idempotent writes keyed on offer_id (UUIDv4 from DISCO) so a retried INSERT is a no-op. Reads are point lookups on offer_id for state checks; the offer reaper does a WHERE expires_at < now() time-bucketed scan over a separate offer_by_expiry(time_bucket, offer_id) table to avoid SELECT * tombstone walks.

Why it exists. Cassandra (or Schemaless on top of MySQL — Uber's actual canonical) was chosen for marketplace state because the workload is write-heavy (5K offer writes/s peak, mostly idempotent INSERTs), single-key ACID is sufficient, and we need cross-region replication that doesn't block on a leader. Postgres-HA was rejected: even with sharded Citus, a write spike on one shard during an event sets up a queueing pile-up that the QUORUM-leaderless model absorbs through hinted handoff.

When it fails. Quorum lost in one AZ during a network partition: W=2 across the surviving 2 AZs still works but read repair lags; if a SECOND AZ blips, both W and R fail closed and DISCO sees quorum_unavailable. Detection: cassandra_quorum_unavailable_rate > 0 keyspace-scoped. Mitigation: explicit 'degraded dispatch' mode lowers reads to LOCAL_ONE (kRing-side reads only) but keeps writes at QUORUM — accept staleness on candidate search, never on offer commit; widen kRing radius and lengthen offer TTL so the matcher tries again on a fresh quorum.

Stream · Marketplace BusKafka 3.7, RF=3, 24 h retention, per-topic acks tuning

Three topics with different durability tiers: (1) loc.heartbeats — every driver GPS sample, ~1.875M msg/s peak, 256 partitions by city_id, producer acks=1 (heartbeats are best-effort; dropping a few is fine because the next sample replaces them in 4 s); (2) dispatch.events — offer state transitions (OFFERED, ACCEPTED, EXPIRED) for downstream surge + analytics + audit, ~5K msg/s, 64 partitions by offer_id, producer acks=all + min.insync.replicas=2 (offer events are the outbox — RPO 0 in-region, that's load-bearing); (3) surge.outputs — windowed multipliers for cache backfill + audit, ~50K writes/s peak, 32 partitions, acks=1. Cross-region replicated via uReplicator at <5 s lag (RPO bound for the dispatch outbox).

Why it exists. The bus is the load-bearing decoupler between the synchronous dispatch path (sub-second) and the windowed surge / analytics path (minutes). It also serves as the durability layer for driver heartbeats (Redis is non-persistent — durability lives here on a 24 h retention) and as the transactional outbox for offer commits (DISCO writes to Cassandra and the bus in the same offer-commit group; if the bus enqueue fails, the outbox-relay worker re-drives from Cassandra CDC).

When it fails. Consumer lag on loc.heartbeats: surge aggregator computes off stale supply, multiplier flickers vs offered price. Detection: consumer_lag_seconds{topic=loc.heartbeats} > 30 for 60 s. Mitigation: surge offer envelope is version-stamped; offers honor the quoted price for offerTtlSec on confirm regardless of refresh. Aggregator backlog drains by horizontal-scaling consumers; partition count chosen so a 2x consumer scale-out is always possible without re-keying. Poisoned-message handling: bus → DLQ (Queue · Poisoned Offer DLQ) routes events that fail consumer parse 3 times to the DLQ; on-call replay tool drains DLQ after fixing the consumer.

Worker · Surge AggregatorApache Flink 1.18, RocksDB state, Beam SDK

Streams loc.heartbeats and dispatch.events from the bus, keyed by h3_r7_cell. Maintains a 1-minute sliding window with 10-second slide. At each slide: counts unique drivers (supply) and rider requests (demand), divides into a ratio, smooths with EWMA over the trailing 3 windows, looks up the cell-specific surge curve, and writes (cell, multiplier, version, computed_at) to the surge cache + the surge.outputs audit topic. Poison messages are routed to the DLQ after 3 parse failures, never blocking the stream.

Why it exists. The surge multiplier is the only piece of dispatch metadata that needs windowed aggregation across all of supply and demand — every other read (driver location, ride state) is a point lookup. Flink owns this because (a) exactly-once semantics matter (double-counting demand inflates surge), (b) RocksDB-backed keyed state at 6M keys × 3 windows is tractable, (c) checkpoint-based recovery means a crash replays from the last consistent window, not from scratch. Spark Streaming was rejected: micro-batch latency floor is 1–2 windows, kills the 'react to a stadium ending' story.

When it fails. RocksDB state grows past TM heap during a deploy + traffic spike: the aggregator stalls; surge multipliers stop refreshing; ETA falls back to the last cached value plus the 60 s TTL window. Detection: flink_state_size_bytes > 80% TM heap AND checkpoint_duration_p99 > 60 s. Mitigation: state TTL on inactive cells (reaped after 5 unobserved windows), checkpoint to GCS not local disk so a TM swap doesn't lose state, manual repartition (re-key by region) when a single TM holds > 1M cells.

External · Push (APNs / FCM)Apple APNs HTTP/2 + Google FCM HTTP v1

Receives a (driver_token, offer_id, expires_at, surge_version) envelope from DISCO and delivers a high-priority push to the driver's iOS or Android device, waking the app to display the offer. APNs/FCM is the wakeup, NOT the transport — the actual offer payload is delivered via the WS the driver app already holds; the push only cues 'there's an offer waiting on the wire'.

Why it exists. iOS aggressively kills backgrounded sockets after ~30 s; without push, half of off-duty drivers would never get an offer. We considered making push the transport itself (envelope in the push payload) — that loses guaranteed-delivery: APNs may drop a push under load, and we'd need an out-of-band ack. Push-as-wakeup + WS-as-transport is the load-bearing belt-and-suspenders.

When it fails. APNs returns 503 for 10 minutes (an actual recurring incident pattern). Detection: push_provider_5xx_rate > 5% for 60 s, paired with offer_no_driver_ack_rate rising. Mitigation: WS-only fallback for foregrounded drivers (most of the active pool) — backgrounded iOS drivers go silent, rider sees longer match time and the matcher widens kRing to compensate. Rider-side UX shifts to 'matching is taking longer' rather than failing closed.

External · Maps & RoutingGoogle Maps Routes API + in-house OSRM fallback

Returns a routing result (distance_meters, duration_seconds, route_polyline) for an (origin, destination) pair. Used by ETA service to compute the base ETA before the DeepETA residual is applied, and by the Driver app for in-trip navigation (out of scope for matching). Treated strictly as a third-party with bounded availability and rate limits.

Why it exists. Building a worldwide routing engine is a separate billion-dollar problem. We pay Google's per-call price for the SLA they guarantee and run an in-house OSRM cluster as a degraded fallback for when Google's API rate-limits us or returns 5xx. Doing matching without routing is theoretically possible (haversine straight-line distance × fudge factor) but the ETA accuracy hit costs us match quality.

When it fails. Google Maps Routes API returns 429 (rate-limited) during a regional traffic spike. Detection: maps_429_rate > 1%. Mitigation: ETA service auto-flips to the OSRM fallback for that region, doubles the per-request timeout (OSRM is slower), and tags the offer envelope with routing=osrm for observability. Riders see a few-second ETA delta (worse on traffic-heavy routes) but the matching path itself never blocks on Maps.

Monitoring · M3 + Prom + AlertmanagerM3 (Uber) for metrics, Prometheus federation for tactical, PagerDuty

Scrapes /metrics from every service (gateway, riderapi, eta, disco, locingest, surgeagg, ring, glb, reaper, outboxrelay) and a sidecar exporter for every stateful tier (geoidx, surge, rides, bus, dlq). Aggregates per-(region, city, h3_r7_cell) so the on-call queries dashboards by cell, not by host. Owns the SLO definitions and the alert routes (page on dispatch availability < 99.95% over 5 min, p99 > 3 s sustained, heartbeat-to-visible > 8 s, offer_acceptance_rate drop > 10pp vs trailing 24h, surge_version_mismatch_on_confirm > 0.5%, offers_routed_to_wrong_owner > 0).

Why it exists. Without an explicit observability tier on the diagram, the SLOs and detection signals scattered across every node's failureMode have no system that actually measures and alerts on them. We considered embedding observability into each service (Datadog agent / OpenTelemetry collector per pod); that's the implementation, but the canonical needs to surface the dependency surface — what gets monitored, what pages, who owns the rotation.

When it fails. M3 ingestion buffer fills during a metrics tsunami (deploy + spike): metrics arrive late, alerts fire 2–5 min after the user impact starts. Detection: m3_ingest_lag_seconds > 60. Mitigation: pre-aggregation at the exporter level (per-host roll-ups) so a 1000-host fleet emits one rollup per service per minute, not 1000; alert rules use 5-min windows so a 2-min ingest blip doesn't suppress detection; tactical Prometheus federation continues to fire on tactical thresholds independently.

Scheduler · Offer ReaperGo cron, leader-elected via etcd lease, 1 active per region

Every 1 second scans the offer_by_expiry(time_bucket, offer_id) Cassandra table for offers whose expires_at < now() AND state = OFFERED, force-transitions them to EXPIRED via a CAS UPDATE on rides, and publishes EXPIRED events to dispatch.events on the bus so DISCO + downstream consumers see the transition. The reaper exists outside DISCO precisely so a poisoned in-memory state machine inside DISCO cannot wedge a rider waiting on a stuck offer.

Why it exists. DISCO drives the offer state machine in-memory; if a DISCO process crashes mid-OFFER (or its state machine wedges on a buggy code path), the offer is stuck OFFERED forever from the rider's perspective. We considered making the rider-side WS reconnect kick the reaper; that couples liveness to the rider being online, which is exactly the case where the rider is most frustrated. External sweeper is the load-bearing liveness guarantee.

When it fails. Reaper itself crashes: stuck offers sit OFFERED until a backup leader picks up the etcd lease (sub-10 s). Detection: scheduler_leader_election_lag_seconds > 10 OR offers_in_OFFERED_age_p99_seconds > 30. Mitigation: 3 standby reapers per region with 5 s lease renewal; alert if zero healthy reapers for >30 s; runbook includes a manual-EXPIRE tool that ops can run from a bastion if all reapers fail.

Worker · Outbox Relay (CDC)Debezium-style CDC tailing Cassandra commit log → Kafka

Tails the Cassandra commit log via a CDC connector for the offer table, identifies rows whose relayed_at is null beyond a 5-second threshold (i.e., DISCO's primary path failed to enqueue to bus), republishes the event to dispatch.events, and stamps relayed_at. The CDC stream is the recovery path that makes the offer-commit outbox correct under partial Kafka failure.

Why it exists. DISCO's primary path is: write Cassandra (durable) → publish bus (best-effort relay). If Kafka is briefly unavailable when DISCO tries to publish, the offer record is durable but the event is lost — downstream consumers (surgeagg, analytics, audit) miss the OFFER. We considered making DISCO retry Kafka indefinitely on failure; that blocks the dispatch return, busting the SLO. The CDC relay decouples durability from delivery.

When it fails. CDC connector falls behind Cassandra write rate (5K offer/s peak): unrelayed events accumulate; downstream consumers see growing OFFER lag. Detection: outbox_relay_lag_seconds > 30 OR cdc_tailer_offset_behind_writes > 10000. Mitigation: 8 connector instances partitioned by hash(offer_id) so a single-connector slowdown doesn't block the rest; alerting fires before lag reaches the 24h Kafka retention floor; operator runbook drains the lag by spinning additional connectors temporarily.

Queue · Poisoned Offer DLQKafka topic dispatch.events.dlq, RF=3, 7d retention, manual drain

Receives offer events (or surge windowed inputs) that surgeagg or other consumers fail to parse 3 consecutive times. Holds them for 7 days on a separate Kafka topic so the on-call has time to identify the schema regression, fix the consumer, and replay. Never auto-replayed into the hot path — replay is an explicit ops action.

Why it exists. Without a DLQ, a poisoned message (e.g. a surge_version that no longer parses after a model rollback) cycles forever in surgeagg's fixed-retries loop. At 5K offer/s a 1% poisoned rate is 50/s into an unbounded retry — within minutes the consumer is pinned on bad messages, lagging the legitimate ones. DLQ gives consumers an exit valve.

When it fails. DLQ depth grows during a sustained schema-mismatch incident: alerts fire on dlq_depth > 1000. Mitigation: alert routes to the consumer team (NOT the dispatch team) so the right engineer triages; replay tool batches by 1000-message chunks with a kill switch; if the schema regression is in the producer (DISCO), rollback is the fix, NOT replay.