Notification System — a worked solution

What each component is for

Client

Two distinct callers share this entry point: (a) an internal service emitting a transactional event ('order shipped', 'ride arriving', 'password reset', '2FA code') via POST /v1/notifications with an Idempotency-Key; (b) a merchant integration POSTing webhook deliveries when this same platform doubles as a webhook fan-out tier. Both expect a 202 Accepted in <100 ms — the actual delivery happens async on the Kafka spine.

Why it exists. Considered modeling internal-callers and merchant-integrations as separate edge nodes. Rejected because their failure shapes are identical (idempotent retry, HMAC-signed body, exp backoff) and the only divergence is auth method — service-mesh JWT for internal, scoped API key for merchants — which we draw at the auth node downstream rather than at the edge.

When it fails. Naïve clients regenerate idempotency keys on retry → ghost notifications. Detection: distinct_keys_per_logical_event_p95 > 1 alerts SDK eng. Mitigation: official SDKs ship the canonical retry helper; broken clients are caught at the gateway via an HMAC-signed request fingerprint mismatch (Stripe pattern: same key + different body → 422).

API Gateway · Public EdgeEnvoy + Cloudflare WAF + TLS terminator

L7 ingress for /v1/notifications and /webhooks/{provider}. Terminates TLS, runs the WAF ruleset (OWASP top-10 + PII rules — reject log payloads carrying card numbers), enforces token-bucket rate limits per tenant BEFORE forwarding so retry storms can't poison the idempotency store, copies the Idempotency-Key header verbatim onto the upstream request and signs the forwarded headers with HMAC so the acceptor trusts tenant_id without re-validating.

Why it exists. Considered putting auth + rate limit inline in the notification acceptor. Rejected because (1) a 5xx in the acceptor would skip the rate limiter and let the storm reach the idempotency store, blowing out conditional-write capacity; (2) per-tenant rate-limit data lives in a single Redis cluster which we'd otherwise ship to every acceptor replica. Stripe's 2019-07-10 retro showed what happens when rate limits live in the wrong layer — the cascade crosses product surfaces.

When it fails. Edge pool exhaustion masks as 'producer says we're slow' but is actually upstream load. Detection: gateway_active_streams, downstream_5xx_rate. Mitigation: drain mode + per-zone failover; circuit-break a misbehaving tenant rather than 503'ing everyone. Cloudflare's Jun 2022 retro warns: anycast misconfig can route traffic to the wrong region — we monitor region_active_set_size and page if a tenant has >1 active region simultaneously.

Auth / IdentityHashed scoped-API-key + service-mesh JWT (SPIFFE)

Validates the Authorization header on every public call: hashed key lookup against a sharded Postgres keys table, scope check (notifications:write vs webhooks:read vs admin:read), per-key revocation list. Internal calls use service-mesh JWT (SPIFFE/SPIRE) and skip the keys lookup. Returns a signed (tenant_id, scopes, allowed_channels) envelope back to the gateway, valid for 60 s.

Why it exists. Considered baking authn into the acceptor. Rejected because (1) a CVE in the acceptor should not break authn for everyone (blast-radius isolation), (2) the 100M-row keys table is hot in cache and doesn't belong on the notification hot path, (3) scoped keys (e.g. notifications-only API keys for marketing tools) are a separate product surface that needs scope validation independent of any single service.

When it fails. Auth down → ALL writes fail-CLOSED 503 (better than fail-OPEN which would let unauth traffic through). Cached scopes survive 60 s then 401. Detection: auth_5xx_rate, scope_cache_miss_rate. Microsoft Teams 2020-02-03 cert-expiry outage (4 h global) is the cautionary tale — cert-manager + alert at T-30 days. Square 2023-09-07 cert-bundle outage is the same lesson at the service-mesh layer.

Service · Notification AcceptorGo + AWS SDK v2 + librdkafka (idempotent producer)

Hosts POST /v1/notifications. For each request: (1) checks the region-lease — if our region is not the active leader for this tenant, return 503 + Retry-After BEFORE touching any durable state (claiming the key or publishing from a standby region would strand a committed message the client was told failed); (2) hashes (tenant_id, idempotency_key, body) into a request fingerprint; (3) attempts a DynamoDB conditional PutItem with attribute_not_exists on the idempotency table — wins → claim; loses → returns the cached response_id; (4) publishes a notif.requested event to Kafka with the idempotent producer (acks=all, enable.idempotence=true) keyed by recipient_id. Returns 202 in <100 ms.

Why it exists. Considered fronting Kafka directly with a thin gateway plus a separate dedup worker. Rejected because the dedupe-then-publish ordering MUST be tight — if dedup happens after publish, a Kafka producer retry creates a dup that the consumer can't dedupe without an end-to-end key. Considered Kafka transactions (exactly-once semantics within Kafka). Rejected because EOS doesn't span Kafka + DynamoDB writes; the cross-store dedupe must live OUTSIDE Kafka. The acceptor is where producer-side dedupe happens; the dispatcher does a SECOND dedupe at send-time as defense-in-depth (Stripe webhook pattern).

When it fails. Idempotency store down → fail-CLOSED 503 (better to lose a request than dup). Kafka producer cluster unreachable → fail-CLOSED 503. Region-lease unreachable (etcd partitioned) → fail-CLOSED 503 (better to refuse than risk both regions sending). Detection: claim_p99 (alert <50 ms 1-min P2), kafka_produce_ack_p99 (alert <200 ms P2), region_lease_check_p99 (alert <30 ms P2). Cloudflare 2022 / Slack 2021 lesson: never fail-OPEN on the fence check — that's the path to double-sends.

Service · Preference / Opt-Out APIGo + Cassandra driver (LWT) + Redis client

Hosts the preference + opt-out write surface — PUT /v1/users/{id}/preferences and POST /v1/users/{id}/opt-outs — and is where the SMS dispatcher's inbound-'STOP' webhook lands. For an opt-out: (1) INSERT ... IF NOT EXISTS (Cassandra LWT / Paxos at LOCAL_SERIAL) into the user's home-region pref-store for linearizable consistency within that cluster — the TCPA primitive (cross-region is async-mirrored, so the pre-send check must read the home region synchronously); (2) invalidate pref:{user_id} in pref-cache so the router re-reads on the next event (invalidate-on-change, not write-through, to avoid the stale-but-fresh race). Distinct from the notification acceptor, which only writes the Kafka spine.

Why it exists. Considered folding preference writes into the notification acceptor. Rejected because the acceptor is tuned for the high-QPS fire-and-forget publish path (DynamoDB claim + Kafka produce), whereas opt-out writes are low-QPS but must be LINEARIZABLE (LWT/Paxos, ~30 ms) — mixing the two couples a rare-but-legally-critical write to the hot path. Separating the plane also keeps the TCPA-exposed code surface small and auditable.

When it fails. pref-store unreachable → opt-out write fails-CLOSED 503 (never ACK an opt-out we didn't durably record — TCPA exposure). Cache-invalidate drop → bounded by the 5 s opt-out TTL on pref-cache. Detection: optout_lwt_p99, optout_write_5xx_rate, pref_cache_invalidate_lag_seconds. Satterfield v Simon & Schuster is the precedent — a missed opt-out is a per-message statutory liability.

KV Store · IdempotencyDynamoDB Global Tables (us-east-1 ↔ us-west-2)

Stores per-Idempotency-Key state for the producer-side dedupe AND per-(notif_id, channel, recipient) state for the dispatcher-side defense-in-depth dedupe. Schema: (PK = sha256(tenant_id, idempotency_key), request_fingerprint, response_id, response_code, response_body_compressed, ttl). Two operations: CLAIM = PutItem with ConditionExpression 'attribute_not_exists(PK)' (fresh) — wins → claim; FetchExisting = GetItem on conflict → returns cached response. Same store, different PK namespace for the dispatcher dedupe.

Why it exists. Considered Redis SETNX. Rejected because GitHub 2018-10-21 retro shows SETNX races during cluster reshards (we'd hit the same on a Redis cluster topology change), and Redis async failover has RPO=5 s — 5 seconds of dedupe state vanishing on AZ outage is enough to leak dupes. DynamoDB conditional PutItem is the AWS Builders' Library 'Idempotency at Scale' standard answer (re:Invent ARC403 2021). Considered Postgres unique constraint — works but cross-region replication via logical-decoding is more operationally complex than DynamoDB Global Tables, which AWS manages.

When it fails. Cross-region replication lag > 5 s (AWS-published worst case 30 s) opens a dedupe-bypass window — a producer retry that lands in the standby region before its idem-row replicates from the home region wouldn't dedupe. Mitigation: regional ownership — each tenant has a HOME region (consistent-hash on tenant_id); the OTHER region 503s for that tenant unless an explicit failover ramp has fired. Defense-in-depth dispatcher dedupe (in-region) catches same-region replays even if cross-region leaks. Detection: dynamodb_replication_lag_seconds (alert >5 s P2, >15 s P1), ConditionalCheckFailed_rate (high = busy), throttled_writes_total (high = capacity issue). Single-row blast radius — never larger than one Idempotency-Key.

Coordinator · Region Leaseetcd 3.5 (5-node Raft cluster across 3 AZ)

Holds a per-tenant 'active region' lease at /lease/{tenant_id} → region_id, with a 30 s TTL renewed by a 10 s heartbeat from the active region. Notif-acceptor and dispatchers query before accepting / sending: if our region != lease holder, fail-CLOSED (acceptor returns 503; dispatcher drops the in-flight message — the active region's dispatcher will pick it up). Operator-flippable for planned failover via a typed runbook.

Why it exists. Considered DNS-based region failover. Rejected because DNS TTL plus client-side resolver caching means both regions can be active for several minutes during a cutover — a guaranteed double-send window for transactional channels. Considered DynamoDB Global Tables for the lease (single-row conditional writes). Rejected because Global Tables replication lag is documented up to 30 s, and the lease IS the safety primitive — the lease store must have STRONGER consistency than the dedupe store. etcd / Consul give us linearizable leases via Raft. Cloudflare Jun 21 2022 and Slack Jan 4 2021 retros are the precedents — fencing tokens for active-region are the *Database Reliability Engineering* (Campbell & Majors, ch. 9) answer.

When it fails. etcd quorum lost (3 of 5 nodes failed) → all writes fail-CLOSED → notification system halts (no fence = no send). Severe SPOF. Mitigation: 5-node cluster across 3 AZs + cross-region observers (operator-driven failover RTO ~10 min); per-tenant cached lease state survives 30 s on etcd unavailability so brief blips don't halt the system. Detection: etcd_has_leader, etcd_proposals_committed_total, lease_renewal_lag_seconds. Trade: we accept full-system halt over double-send risk (lower-cost incident than TCPA / merchant-trust damage).

Stream · Notification SpineApache Kafka 3.7 (KRaft mode, 24 brokers, 3 AZ)

Per-region Kafka cluster carries every notification through the system — the tenant's home-region cluster is the spine of record, async-mirrored to the standby region via MirrorMaker 2 so a fenced cutover resumes from a local copy. Topics: notif.requested (post-acceptor), notif.{push,email,sms}.queued (post-router, per-channel for bulkheading), notif.{push,email,sms}.retry (delayed-retry topics with timestamp-keyed exp backoff), provider.feedback (webhook receiver), notif.delivered (audit). Partitioned by hash(recipient_id) so per-recipient ordering is preserved (a 'started' notif precedes the 'completed' for the same trip).

Why it exists. Considered SQS / RabbitMQ. Rejected because (a) SQS FIFO groups cap at 300 msg/sec/group which can't carry our peak 92 K QPS without painful sharding gymnastics, (b) replay-from-offset for bug-fix backfills is impossible with SQS's 14-day point-of-no-return retention, (c) multi-consumer fan-out (push + email + sms + audit + analytics all reading the same notif.requested) wants Kafka's pub-sub model, not point-to-point. Per Uber's Push Platform, DoorDash's notification platform, LinkedIn ATC, and Slack's Job Queue — Kafka is the production answer.

When it fails. Single broker loss → controller re-elects leaders for affected partitions, ~5 s gap on the affected partition's writes. min.ISR=2 → single-broker loss never blocks; two-broker loss blocks writes (correct — better than risking data loss). Hot partition (10× QPS on one recipient — viral content / mass corporate user) lights up monitoring; mitigation is partition-level rate limiting + recipient-group coalescing at the dispatcher. Consumer-group rebalance during deploys is the LinkedIn KIP-429 lesson — cooperative rebalancing turned a 30-60 s STW into seconds; static membership (group.instance.id) makes restart-in-place rebalance-free.

Service · Router & RendererJava + Kafka Streams + handlebars/Mustache templates

Consumes notif.requested. For each event: (1) fetch user preferences + device tokens from preference cache (miss → Cassandra); (2) apply frequency cap (LinkedIn ATC rule engine — e.g. max 5 notif/user/hour for non-critical); (3) apply quiet-hours filter in user-local timezone (NOT UTC); (4) render the template per channel (handlebars — Twitter '{firstName}' bug prevention via golden-dataset render-diff in CI); (5) for each channel that survives filtering, publish a notif.{push,email,sms}.queued event back to Kafka with the rendered payload. Stateless — all state is in pref-store/cache.

Why it exists. Considered doing routing inside the channel dispatchers. Rejected because that path makes every dispatcher re-render and re-fetch preferences — 3× the read load on pref-store and N× the duplicated render code paths. Centralizing routing means ONE preference fetch per business event regardless of how many channels fire. Considered making this a Kafka Streams app (stateful join). Rejected for operational simplicity — stateless service + cache hits the same 90%-of-events case at lower complexity. LinkedIn Air Traffic Controller (Espresso + Couchbase + Samza dispatcher) is the canonical reference; we adapted the shape to Cassandra + Redis + plain Kafka consumers.

When it fails. Pref-cache cold restart → all reads hit Cassandra → LOCAL_QUORUM saturates → render p99 climbs from 30 ms to 300 ms. Detection: pref_cache_hit_ratio < 0.85 P2 page; render_p99 > 100 ms P2. Mitigation: warmup job replays last 30 min of events on restart; circuit breaker fast-fails when Cassandra > 200 ms p99 (degrade to 'best effort default preferences' for non-critical channels). Stateless — scaling is horizontal, no local state to lose. Malformed-template deploy (Twitter '{firstName}' literal) prevented by golden-dataset render-diff in CI + canary rollout.

Cache · PreferencesRedis 7 cluster mode, 16 shards

Caches pref:{user_id} (preferences blob, TTL 60 s) and freq:{user_id} (rolling-window frequency counter, TTL 1 h). Populated lazily by the router on miss; INVALIDATED (not written-through) by the user-API when the user changes preferences — invalidate-on-change avoids the double-write race that gives stale-but-cache-fresh data.

Why it exists. Considered putting the cache inside the router process (in-memory). Rejected because (1) cache hit ratio depends on cross-replica sharing — in-memory caches partition by replica and give 1/N hit ratio at our 16-replica fleet, dropping our effective hit rate to ~6 %; (2) opt-out invalidation needs to touch ALL caches atomically, which is easier with a centralized Redis. Considered making Cassandra the only store. Rejected because at 100 K events/sec peak that's 100 K Cassandra LOCAL_QUORUM reads/sec — Cassandra prefers <50 K reads/sec/cluster at this footprint without expensive horizontal scaling.

When it fails. Cluster cold restart → 100 % miss → router falls through to Cassandra → see pref-store failure mode (saturates LOCAL_QUORUM reads). Single shard down → 1/16 of users go cold; router degrades to Cassandra for those slot groups. Detection: hit_ratio, cluster_state, slot_health. Mitigation: warmup job replays preferences for the top-1M most-active users on startup. NEVER trust the cache for opt-outs — those go straight to pref-store via Cassandra LWT (TCPA exposure if the cache leaks).

NoSQL DB · Preferences & TokensCassandra 4.1, RF=3, LOCAL_QUORUM

Source-of-truth for: (a) per-user preferences (channels enabled, quiet hours, opt-outs, frequency caps); (b) device-token registry (one row per (user_id, app_id, device_id) with token + last_seen_at + status); (c) suppression lists (bounced emails, complained addresses, blocked numbers). Read-mostly (~95 % reads from the router hot path, ~5 % writes from webhook GC + user opt-out flows).

Why it exists. Considered Postgres + read replicas. Rejected because at 200M users × 2.5 devices × 3 apps the row count breaks single-shard Postgres; cross-shard reads on the router hot path would need a fan-out which Cassandra avoids via partition key = user_id. Considered DynamoDB. Rejected because per-user query patterns include 'all my devices for app X' which is a sort-key range scan in DynamoDB (provisioned-RCU expensive at this scale) or a Cassandra clustering-key scan (cheap). Considered Aerospike — works but our team owns Cassandra ops. The Monzo 2019-07-29 retro on auto_bootstrap is the cautionary tale we learn from for cluster scale-out.

When it fails. AZ loss → quorum still met (2 of 3 replicas survive). Region loss → operator-driven cross-region promote (we deliberately don't auto-promote because cross-region cassandra-stream-mirroring lag varies under load). Hinted handoff handles brief replica unavailability. Monzo 2019-07-29 lesson: never relax to consistency=ONE on writes that must be authoritative — for opt-outs we use LWT (Paxos) and ACCEPT the latency cost. Detection: hinted_handoff_queue_depth, read/write timeout rate, cluster status, optout_replication_lag_seconds (P1 alert). Single-partition blast radius — 1/N users.

Worker · Push DispatcherGo + APNs HTTP/2 client + FCM Admin SDK

Consumes notif.push.queued from Kafka. For each message: (1) defense-in-depth dedupe — conditional PutItem on idem with PK=(notif_id, channel='push', recipient_token); (2) check the region-lease — if not the active region for this user, drop the message (active region's dispatcher will pick it up); (3) send via APNs HTTP/2 (apns-priority based on notification class) or FCM HTTP v1 (multicast up to 500 tokens for fan-out coalescing); (4) on success, publish notif.delivered to Kafka for audit. Per-provider circuit breaker with 50 % error threshold over 30 s window.

Why it exists. Considered one polyglot dispatcher for all channels. Rejected because the Twilio Jul 18 2023 retro shows EXACTLY what happens when channels share a thread pool — Twilio's degradation took down push and email in teams that didn't bulkhead. Per-channel dispatchers give us the bulkhead. Considered making this a Lambda. Rejected because per-call cold-start tail (200-800 ms) violates push p99 budget, and the long-lived APNs HTTP/2 connections (worth keeping warm — TLS handshake is 50 ms) want a long-lived process.

When it fails. APNs/FCM 5xx storm → circuit opens → messages divert to notif.push.retry (delayed Kafka topic with timestamp-keyed scheduling). Discord 2018 push-retry-storm + AWS Builders' Library precedents. Detection: apns_5xx_rate, circuit_breaker_state, push_p99, push_dispatcher_pool_saturation. Token churn: APNs returns 410 Unregistered (and FCM returns UNREGISTERED) SYNCHRONOUSLY on the dispatcher's own HTTP/2 send — the dispatcher detects it inline and emits a token-GC event that deletes the token from the pref-store (no inbound webhook; APNs/FCM don't POST callbacks). Daily Spark job over pref-store reaps tokens with last_seen > 90 d as backstop. Per-channel bulkhead means a push outage doesn't drag SMS or email down.

Worker · Email DispatcherGo + AWS SES SDK + SendGrid v3 client

Consumes notif.email.queued. For each message: (1) defense-in-depth dedupe; (2) region-fence check; (3) suppression-list lookup against pref-store (skip if email is suppressed due to bounce/complaint); (4) route to the right IP pool — transactional pool (high reputation, dedicated IPs) vs marketing pool (separate reputation domain); (5) send via SES; on Throttling, vendor-failover to SendGrid (manual approval — too easy to make worse); (6) publish notif.delivered to Kafka.

Why it exists. Considered making the email path a thin Lambda over SES SendEmail. Rejected because (1) suppression-list lookups want a long-lived connection to pref-store (Lambda spins one up per cold start), (2) IP-pool routing is non-trivial state, (3) SES Throttling fast-fails benefit from a circuit breaker held by a long-lived process. Considered SES alone. Rejected because we need vendor failover — when SES has a regional outage we route to SendGrid; relying on one provider for the most-trafficked channel is the SPOF that retro pages get written about (e.g. Mailgun's 2018 outage took down Slack invites).

When it fails. SES Throttling → circuit opens, messages divert to notif.email.retry. Bounce-rate spike (>5 %) → SES auto-suspends sending → vendor failover to SendGrid (manual approval). SendGrid IP-reputation crash from a bad campaign — quarantine campaigns at >2 % bounce in 1 h. Detection: bounce_rate_15m, complaint_rate, ses_5xx_rate. Per-channel bulkhead means an email outage doesn't drag push or SMS down.

Worker · SMS DispatcherGo + Twilio Programmable SMS + Sinch client

Consumes notif.sms.queued. For each message: (1) defense-in-depth dedupe; (2) region-fence check; (3) carrier lookup (E.164 normalization + carrier prefix) to pick the right Twilio messaging-service-SID for that destination's regulatory regime (US 10DLC, UK shortcode, IN DLT-registered, EU sender-ID); (4) per-MSISDN rate-shape (Twilio long-code = 1 MPS, short-code = 100 MPS, A2P 10DLC tiers up to 75 MPS); (5) SYNCHRONOUS opt-out check against pref-store (TCPA compliance — never trust the cache); (6) send via Twilio Messages API; on 503, vendor-failover to Sinch.

Why it exists. Considered putting SMS into the same dispatcher as push. Rejected because SMS rate-shaping is fundamentally different — push has a pool of HTTP/2 connections, SMS has a pool of SENDER NUMBERS each with its own carrier-imposed cap. The number-allocation logic is non-trivial (carrier-prefix → messaging-service-SID lookup) and doesn't belong in a polyglot dispatcher. Lyft's published 'SMS Reliability' post documents exactly this split (2019).

When it fails. Twilio 503 → vendor failover to Sinch (manual + automatic depending on incident length). Twilio's Jul 18 2023 incident is the precedent (global SMS queue). Detection: twilio_5xx_rate, sinch_5xx_rate, sms_p99_carrier_ack. Per-channel bulkhead (separate consumer group + topic + dispatcher pool) means a Twilio outage doesn't take down push or email. Carrier-filtering escalation: if 30007 rate spikes, auto-quarantine the template and page content team. Opt-out replication lag → fail-CLOSED with retry-after; legal cost of one missed opt-out >> latency cost.

External · Provider GatewaysAPNs HTTP/2 / FCM HTTP v1 / SES / SendGrid / Twilio / Sinch

Receives push, email, and SMS messages from our dispatchers and delivers to the recipient's device, mailbox, or carrier. Sends asynchronous webhook callbacks to our webhook receiver for delivery state changes (delivered, bounced, complained, unregistered).

Why it exists. We don't run our own carrier interconnect or MTA fleet — at this volume, multi-vendor providers (Twilio + Sinch for SMS, SES + SendGrid for email, APNs + FCM for push) give us higher delivery success than home-rolled would, plus they handle the regulatory minefield (10DLC TCR registration, GDPR, DKIM key management, APNs cert provisioning). Considered self-hosted Postfix + carrier SS7 connections. Rejected because that path needs an 18-month build, ~$5M/yr ops cost, and the regulatory risk (a misrouted SMS to a country we lack a sender-ID in is an immediate carrier ban) is the kind of thing a payment-system retro is written about.

When it fails. Single provider down → vendor failover (preferred). Multi-provider correlated failure (rare — a regulatory or DNS-level event) → channel goes dark; circuit breakers fast-fail; messages divert to retry topic. Detection: per-provider 5xx rate, delivery success rate (NOT what the provider reports — the GAP between provider-reported-success and device-acked-success is the gray-failure tell flagged by Uber and LinkedIn). Twilio 2023, FCM Aug 2022 outages both hit this pattern.

Service · Provider Webhook ReceiverGo + AWS SDK + per-provider signature verification

Hosts /webhooks/{ses,sendgrid,twilio,sinch} — the providers that actually POST callbacks. APNs and FCM do NOT send delivery/unregister webhooks: their 410 Unregistered / UNREGISTERED comes back SYNCHRONOUSLY on the push-dispatcher's own HTTP/2 send, so push-token GC is initiated at the dispatcher, not here. For each inbound callback: (1) verify the provider's signature (SNS RSA-certificate signature for SES events, ECDSA P-256 for SendGrid's Signed Event Webhook, HMAC-SHA1 X-Twilio-Signature for Twilio); (2) dedupe against a separate idempotency PK = (provider, provider_event_id) — 30-day TTL covers SES/Twilio replay windows which can stretch up to 30 d; (3) classify: delivered / bounced / complained; (4) for hard bounces / unsubscribes, update the suppression list (and delete the dead email/SMS address row); (5) publish provider.feedback to Kafka for downstream consumers (audit, suppression-list updater, analytics).

Why it exists. Considered routing provider webhooks straight into the API gateway. Rejected because providers' source IPs vary, signature verification is per-provider (different algorithms), and the rate-limit profile is different (providers can dump 10 K webhooks/sec on a delivery batch — completely different from our per-tenant 200/sec ingest cap). Centralizing into a dedicated receiver lets us tune rate-limits and signature checks per provider. Also, the receiver MUST be CRITICAL-AVAILABILITY — if it's down, providers retry for hours and our token-GC stalls (apns_5xx_rate climbs from a flood of dead tokens).

When it fails. Webhook receiver down → providers retry for hours; we backfill once it's back. Token-GC stalls → apns_5xx_rate climbs; mitigation is the daily Spark sweeper over pref-store that reaps last_seen > 90 d tokens regardless of webhook state. Detection: provider_webhook_5xx_rate, signature_verify_fail_rate, token_gc_lag_seconds. SES bounce surge (>5 % → SES auto-suspends sending) is a P1 — paged immediately.

SQL DB · Notification HistoryPostgres 16 + Citus, 64 hash shards by recipient_id

Append-only history of every notification: (notif_id, tenant_id, recipient_id, channel, template_version, sent_at, provider, provider_message_id, status, attempt_count, opt_in_proof_id). Powers /v1/notifications/{id} status (a merchant polling 'did my notification deliver?'), the per-user history UI ('your notifications'), and compliance audits — TCPA / GDPR — 'show me every SMS we sent to this number with timestamp + opt-in proof'.

Why it exists. Considered ClickHouse for analytics + nothing for OLTP queries. Rejected because the /v1/notifications/{id} status endpoint is sub-100 ms p99 (a merchant polling) — ClickHouse doesn't do single-row OLTP at p99. Considered keeping audit in Cassandra alongside preferences. Rejected because audit queries cross-reference opt-in proofs with delivery logs (legal compliance: prove we had consent at send time) and that's a SQL JOIN — Cassandra discourages JOINs and forces app-level fan-out which is slow and error-prone. Sharded Postgres + Citus is the production choice (Stripe, Square Books, Monzo all use this pattern for audit logs).

When it fails. Single shard down → 1/64 of recipients see status-endpoint 503; sends still happen (writes are async via Kafka consumer — the audit write isn't on the critical path). Detection: shard_health, replica_lag_p99, audit_consumer_lag. Replication lag spike: status endpoint reads from leader for fresh data (RYW); analytics reads from follower (eventual). Cold-tier S3 query latency is fine for compliance subpoenas (hours acceptable).