Ticketmaster / Hotel Booking — a worked solution

What each component is for

ClientReact (web) + iOS/Android native

Serves three flows: (a) browse / search results, (b) hold a seat / room for N minutes, (c) confirm + pay. Mints the client-side idempotency key (a UUIDv7 per Pay intent — its embedded timestamp derives the key's daily partition in idempotency_keys) and re-sends it on every retry of the same intent. Holds the waiting-room token in localStorage / Keychain and presents it on every subsequent call.

Why it exists. It's the user. Listed explicitly because the client owns the idempotency key — the alternative (server-minted key) breaks on the exact retry it's meant to defend against: when the user mashes 'Pay' and the response packet is dropped, only the client can supply the same key on retry; a server-minted key would be regenerated and the dedup fails.

When it fails. User loses connectivity mid-confirm. Client retries with the same Idempotency-Key when the network returns; payment-svc serves the cached response. If the user kills the app, the hold expires server-side; on next launch the client sees 409 hold_expired and routes back to the seat picker.

CDN · Bot EdgeFastly + Cloudflare Bot Management

Public IP for every request. Caches search/static (TTL 60s with stale-while-revalidate), routes onsale traffic to the Waiting Room before it can reach the gateway, and runs the bot-manager scoring (device fingerprint + IP-ASN reputation + behavioral signals). Drops 80–98% of bot traffic at the edge before it consumes origin capacity.

Why it exists. Need a tier that absorbs read load AND blocks bots before the inventory tier sees them. Rejected 'rate-limit at the gateway' because by then the bot has already consumed a TLS handshake, a token validation, and a service hop — Queue-it's published numbers show one onsale blocked 2M+ bots (98% of traffic); at gateway-tier cost that's 2M wasted requests through 4 hops.

When it fails. Fastly PoP outage (the 2021-06-08 49-min global outage is the cited precedent). Origin sees a 20–50× spike as the edge cache evaporates. Mitigation: DNS failover to Cloudflare CDN with a 5-min TTL on the public name (the load-bearing DR knob — anycast PoP recovery is not a failover we control); origin sized for ≥3× steady-state for ≥60 min.

Service · Waiting RoomAWS Lambda + API Gateway (deliberately isolated from main fleet)

Holds users in a fair queue when demand >> capacity. Pre-onsale: enrolls users into the room. T-0 (onsale): randomly shuffles the queue (fairness-by-lottery, not arrival time) and starts admitting at the rate the inventory tier can absorb. Mints a signed admission token (HMAC, 5-min TTL) for each admitted user and stores it in the Token Store. The token is required by every subsequent call into the inventory tier.

Why it exists. Need to throttle 3.5M concurrent users into ~100 concurrent shoppers per inventory unit. Rejected 'rate-limit at the gateway' because rate-limit returns 503 with no fairness — users mash refresh, the system becomes a retry storm, and arrival-time bias gives bots the win every time. Ticketmaster SmartQueue's published 13B-bots-blocked-across-17K-events stat is the cost of NOT having this tier. Following SeatGeek's published pattern: this is a deliberately-isolated AWS Lambda + DynamoDB + Fastly stack that does not share fate with the inventory tier — the thing protecting the thing on fire cannot share fate with the thing on fire.

When it fails. Waiting Room own outage: cascades into the main fleet because traffic floods to the gateway. Mitigation: stack isolation (SeatGeek's published lesson — separate AWS account, separate VPC); the inventory tier's emergency-shed mode at the LB returns 503 with Retry-After and a 'currently in maintenance' page. Detected by wr.lambda.5xx > 1% for 60s and gw.unauthenticated_rps > 5x baseline.

KV Store · Waiting Room TokensDynamoDB (on-demand) with TTL attribute

Two item types in one table. (1) token items: active admission tokens minted by the Waiting Room — keyed by token_id, attributes (event_id, user_id, issued_at, expires_at, used_steps[]). Hot path: API Gateway GETs the token on every authenticated call. DynamoDB's native TTL attribute auto-expires tokens — no janitor job. (2) rate-limit items: sliding-window counters keyed by (user_id, window_minute) and (ip_asn, window_minute) — the gateway atomically increments and reads back the count to enforce the 60-holds/min and 6-confirms/min global ceilings (24 gateway replicas cannot enforce a global ceiling from local memory).

Why it exists. Need a globally-replicated, low-latency token store that survives a region failure. Rejected Redis because Redis Cluster has 5s RPO on async failover (config'd as redis-cluster preset) — losing 5s of recently-minted tokens during a regional cutover means the queue admits N users who can't reach the inventory tier. DynamoDB Global Tables replicates asynchronously across 3 regions (multi-active, last-write-wins, typically sub-second lag — there is no synchronous multi-region write mode). We pick it anyway because that sub-second replication RPO is far tighter than Redis's 5s failover window, LWW safely resolves concurrent multi-region token writes, and the admission token is idempotent — so a token lost in the sub-second window during a regional cutover just makes that user re-request (no double-admit).

When it fails. DynamoDB regional outage (us-east-1 2021-12-07). Other regions stay live; admission decisions stall ~5s while clients fail over via Route 53 health checks. Critical: do NOT route admission to a different region without invalidating in-flight tokens — that's how SeatGeek's published runbook covers it.

API GatewayEnvoy + custom auth filter

Validates the waiting-room admission token (HMAC + DynamoDB lookup) on every authenticated call, applies per-user / per-IP rate limits, terminates TLS, and routes to the right backend (search-svc, hold-svc, confirm-svc). Strips raw client headers, injects internal correlation IDs, and enforces a global X-Idempotency-Key requirement on every POST to /confirm.

Why it exists. Need ONE place to enforce token validation, rate-limit, and idempotency-key presence across 4+ backends. Rejected 'each service auths itself' because every backend would re-implement token validation; drift would let bots through on the service that updated late. Envoy's per-route rate-limit + ext_authz integration is the published pattern; idempotency-key gating happens here so a missing key never reaches the confirm service.

When it fails. Bad deploy to Envoy config (the Cloudflare 2022 retro pattern). Mitigation: progressive rollout (1 → 5 → 25 → 100% over 30 min), automatic rollback on 5xx rate > 0.5% for 60s. Detected by gw.5xx_rate and gw.request_duration_p99 against the dual SLO window.

Service · SearchJava + Spring + OpenSearch client

Handles GET /search, GET /hotels/:city/:date, and GET /events?q=.... Reads from Search Index (Elasticsearch), applies business ranking (boost active inventory, demote sold-out), runs A/B tests, and returns paginated JSON ~20–50KB per page. Stateless. Reads (not writes) availability hints from the Hold Store on the detail-page enrichment path.

Why it exists. Need a ranking + business-logic layer that's NOT in the inventory tier's path. Rejected 'query Elasticsearch directly from the gateway' because ES queries lack our business ranking (active inventory boost, sponsored placement, seasonal promotions), A/B test machinery, and result deduplication across overlapping searches; pushing that into ES script-fields gives you 10× the latency on the same node count.

When it fails. Bad deploy ranking model returns garbage results. Detected by search.zero_result_rate > 5% and search.click_through_rate.delta < -20%. Mitigation: feature-flagged ranking, instant rollback via flag flip — does NOT require a redeploy.

SearchElasticsearch 8 (managed: Elastic Cloud)

Inverted index over hotels, events, properties, dates. Faceted search (city + date-range + price + amenities), geo-search, full-text. Refreshed every 1s from the inventory primary via CDC → Kafka → indexer. Returns ranked document IDs + denormalized display fields (name, price, image URL, available-flag-as-of-1s-ago).

Why it exists. Need full-text + multi-dimensional faceted search at 100K QPS. Rejected 'search from Postgres' because GIN indexes + full-text + geo + range filters on a single Postgres query at 50–100K QPS is infeasible at our row count (~100M active properties × 365 dates = 36B unique key combinations); specialized inverted index gives sub-100ms p99 where Postgres gives 5–30s. Following Skyscanner's published pattern: ES as a data service for hotel offers.

When it fails. Cluster red status — one shard's primary + all replicas down (the Stripe 2019-07 gray-failure shape). Detected by es.cluster_status != green for 30s and es.indexing_lag_seconds > 30. Mitigation: snapshot-restore from S3-backed snapshot; meanwhile Search Service falls back to a small in-process cache of last-known-popular results (degraded UX, NOT a hard failure).

Service · HoldGo + Redis client (redis/go-redis)

Owns the hold lifecycle. POST /holds runs a Lua script in the Hold Store that atomically: (1) checks the seat/room/date isn't currently held; (2) writes the hold record keyed by hold:event:E:seat:S (or hold:property:P:date:D) with a 5–15 min PX TTL and the user's holder-token; (3) writes a pointer key hold:id:<hold_id> → the seat key with the same PX TTL so release/confirm can locate the hold from the hold_id; (4) returns the hold ID + holder-token. DELETE /holds/:id runs a Lua check-and-delete that verifies the holder-token before removing. GET /availability/:event/:seat reads the same key. Subscribes to Kafka bookings.confirmed and DELs the confirmed hold's key (holder-token-checked Lua) — the outbox-driven release; the PX TTL is the backstop.

Why it exists. Need a policy + observability layer in front of Redis. Rejected 'client writes directly to Redis' because Redis Lua scripts can't run per-user quota checks, can't emit structured hold.placed audit events for the abuse pipeline, and can't enforce the rate-limit semantics consistently across web/mobile/3rd-party — Hold Service runs those policies and emits the event, Redis runs the lock.

When it fails. Hot-event hot-shard meltdown — one event has 10–17K hold-attempts/sec and ALL of them write to the Hold Store. Detected by redis.shard.write_qps.stddev_ratio > 8x (the hot shard climbs while peers idle). Mitigation: pre-shard the event into seat-block sub-shards (Section 105 rows 1–10 on shard A, 11–20 on shard B); the hot-event escape valve re-routes to a dedicated event-local Redis cluster when the predicted heavy-hitter sketch fires.

Cache · HoldsRedis 7 (cluster mode)

Authoritative store for active holds — Redis keys with TTL ARE the inventory lock. Per-seat / per-(property, date) keys, value carries the holder-token and metadata. Hot path: 50–100K SET NX PX ops/sec at peak. Built-in PX expiry means abandoned holds free themselves with NO janitor job. AOF persistence ON (everysec) so a single-node crash loses at most 1s of holds — and HLL/Redis hold-token semantics make a lost hold a free hold (idempotent: re-creating the same hold is fine).

Why it exists. Need a contended atomic write at 50–100K QPS with built-in TTL. Rejected 'store holds in the inventory DB' because Postgres single-row INSERT ceiling is ~2K/s on strong hardware (research: capacity § 'row-level pessimistic locking breaks at ~1–2K writes/sec') — 50–100K holds/sec would melt the WAL fsync ceiling. Rejected DynamoDB (slower per-op p99 at this throughput, expensive at this volume — ~$5K/day at peak vs ~$200/day for Redis). Redis cluster's per-slot atomic Lua + native PX expiry is what makes hold-then-confirm tractable.

When it fails. Primary shard fails — RPO=1s of recent holds lost. Idempotent semantics make re-creating those holds safe: client retries and the SET NX succeeds because the lost key is gone. The user-visible hand-wave: a user who held seat 14C on the failed primary 1s before crash reads hold_not_found from the promoted replica and may race another user for the same seat during the 1s gap. The booking-row UNIQUE constraint catches the duplicate confirmed booking (defense in depth) but the UX is 'I held that, now someone else got it.' Mitigation we accepted (vs. sync replication at 2-3ms/write cost): bounded blast radius (1 shard ÷ 16 = 6% of holds during the gap), client-side 'hold lost — pick a new seat' UX prompt. Critical: a noeviction OOM IS a hard outage — redis.used_memory_pct > 85 pages immediately because new holds 503 (vs. evicting existing holds, which would silently double-book).

Service · ConfirmJava + Spring + Stripe SDK

Owns POST /confirm. The orchestrator: (1) claims Idempotency-Key in the Inventory DB's idempotency_keys table with a unique constraint, in its own txn (fails fast on replay → returns cached response); (2) verifies the hold still lives in the Hold Store via a Lua check; (3) calls the Fraud Service synchronously (≤200ms, fail-open on circuit-breaker — see fraud node); (4) calls the Payment Provider with the same idempotency-key forwarded; (5) on charge success, opens a SERIALIZABLE Postgres transaction that INSERTs the booking row, INSERTs the outbox row for fan-out, and UPDATEs the idempotency row with the cached response — the Redis hold release is driven by that outbox event (CDC → Kafka → hold-svc), with the PX TTL as backstop; (6) returns 201 with the booking ID. Saga compensation path (5b): if Stripe charge succeeds but the inv-db SERIALIZABLE txn fails (deadlock, primary failover at the wrong instant), publish bookings.failed_post_charge straight to Kafka with acks=all carrying {charge_id, idempotency_key, user_id} — NOT via an outbox row on the shard that just failed (the failover case would share fate with it). At-least-once is safe: Compensation Worker dedups on idempotency_key, and the PSP webhook (charge.succeeded with no matching booking after N minutes) is the durable backstop if confirm-svc dies before publishing. Compensation Worker picks it up and issues a refund — see compensation-svc.

Why it exists. Need ONE atomic boundary that ties the payment authorization to the booking commit, with idempotency on top. Rejected 'client calls Stripe directly + then calls our DB' because the user could be charged but the booking write could fail — and there's no compensating undo because the charge has already cleared Stripe's books; you'd owe the user a refund out-of-band. The dual-write divergence is the classic 'distributed transaction we don't actually want.' Following Stripe's published pattern: the confirm service owns the idempotency contract and the booking write is gated on a durable claim.

When it fails. Payment provider hangs (the cited Visa Europe 2018 retro pattern — provider returns no error, just hangs). Detected by payment_svc.outbound.latency.p99 > 5s for 3m AND payment_svc.http_client.pool_in_use_pct > 90. Mitigation: aggressive timeout (3–5s NOT the default 30s), circuit breaker on payment client, separate connection pools per provider, dual-provider fallback (Stripe → Adyen) via the abstraction layer.

SQL DB · Inventory + IdempotencyPostgres 16 (Patroni + Citus sharding, AWS RDS or self-managed)

System of record. Three tables in one shard: bookings (the durable booking row), outbox (events to publish post-commit), idempotency_keys (client-supplied key + cached response, unique constraint). The idempotency claim runs in its own txn BEFORE the charge (it gates the flow); the post-charge SERIALIZABLE txn INSERTs booking + outbox and UPDATEs the idempotency row with the cached response; the outbox is published asynchronously by CDC Relay. Sharded by event_id (ticketing) / property_id (hotels) so a hot event lives on one shard and doesn't bleed to peers.

Why it exists. Need a system-of-record that supports (a) ACID across booking + idempotency + outbox in one txn, (b) horizontal sharding by event/property so hot events don't melt the global write tier, (c) standard SQL escape hatches for ops (top 100 fraud-flagged bookings, schema migrations, point-in-time restore). Rejected Cassandra LWT (4-round-trip Paxos per write = ~10× latency at this contention; published Monzo 2019 retro shows the operational cost). Rejected Spanner (cross-shard txn is fine but +50% per-write latency and 3× cost; we don't need geo-replicated writes). Following Booking.com's published massively-horizontal sharded MySQL pattern and Brandur's published Postgres-based idempotency-key implementation.

When it fails. Single-shard primary failure (the Stripe 2019-07 retro shape — gray failure cascades across services sharing the shard). Detected by pg.shard.up != 1 and pg.replication_lag_p99 > 2s. Mitigation: Patroni auto-failover to in-AZ sync follower — AZ-RTO = 30s (that 30s is a write outage for 1/16 of events); idempotency keys make retries safe post-promotion. Regional-RTO = 10 min, operator-promoted to the cross-region DR follower (the GitHub 2018-10-21 split-brain MySQL retro is why we do NOT auto-promote on region failure). Note: a us-east-1 region outage drops the inventory primary entirely; this canonical scopes inventory to one regulatory geography — for GDPR / India-DPDP a separate inv-db-eu would be needed, with a gw routing rule on property residency.

External · Payment ProviderStripe + Adyen (dual-provider abstraction)

Authorizes + captures the charge. Confirm Service calls POST /charges with Idempotency-Key: <client-uuid>; Stripe persists the first response (status + body) keyed by it and replays it for 24h (Stripe's published contract). Returns charge_id on success; we attach it to the booking row.

Why it exists. We don't move money ourselves — PCI scope, settlement, and fraud-scoring belong with the PSP. Rejected building our own payment rail because PCI Level 1 compliance is a 6–12 month audit cycle per acquirer + per card scheme + recurring penetration tests; using Stripe drops us to PCI SAQ-A scope. Dual-provider (Stripe primary, Adyen warm fallback) is the cited mitigation for the Visa Europe 2018-class single-provider outage.

When it fails. Provider hard outage (Stripe down). Detected by payment_svc.provider=stripe.error_rate > 50% for 60s. Mitigation: automatic failover to Adyen; holds keep being created (Hold Store still works); confirms route to Adyen — published UX: 'Pay with backup processor' banner. If BOTH providers are down: auto-release-hold mode at 60s lock-up so inventory recycles instead of locking up under un-confirmable holds.

Worker · CDC RelayDebezium + Kafka Connect

Tails the Postgres logical-replication slot on each Inventory DB shard. Reads the outbox table rows (and only the outbox — filtered via Debezium's outbox-event-router SMT), decodes them, publishes to the right Kafka topic (bookings.confirmed, holds.released, etc.), and commits the Kafka offset before advancing the WAL slot. One Debezium connector task per shard (16 shards → 16 tasks spread across the 6 Kafka Connect workers; Connect rebalances tasks to surviving workers on failure).

Why it exists. Need a way to atomically tie the booking commit to the downstream Kafka publish. Rejected 'confirm-svc dual-writes to Postgres AND Kafka' because dual-write is NOT atomic — the DB commits and then the Kafka publish fails (or vice-versa), and you have permanent divergence with no known reconciliation. The transactional-outbox pattern (Confluent's published course) is the only known-correct way: write to the outbox in the SAME transaction as the booking, then a separate CDC relay publishes from the durable log.

When it fails. Relay stalls (consumer crash, disk pressure, Kafka network blip). WAL slot grows; disk pressure on the inventory primary threatens write availability. Detected by pg.replication_slot_lag_bytes > 10GB (P1) and oldest_unpublished_outbox_age_seconds > 30 (P2). Mitigation: scale relays horizontally per shard; if catastrophic, swap to a fresh slot + replay from WAL archive offset (Debezium's published recovery playbook).

Stream · Event BusKafka 3.7 (Confluent / MSK)

Durable event bus for everything that happens post-booking. Topics: bookings.confirmed (P0 priority, partitioned by booking_id), bookings.canceled, holds.released, inventory.changed (for search-index refresh), partner.sync (per-partner priority lanes). 7-day retention enables replay after a consumer bug or schema mistake.

Why it exists. Need a durable, replayable event log between the inventory write and N consumers. Rejected RabbitMQ because once a message is acknowledged it's gone — reprocessing after a consumer bug requires reproducing events from the source-of-truth, which leaks the outbox layer's responsibility back into application code. Kafka's 7-day retention + offset semantics let us replay a downstream consumer's window without any application changes.

When it fails. Single broker failure — automatic leader re-election within 5s (Kafka's published RTO). Detected by kafka.under_replicated_partitions > 0 AND bookings.confirmed.publish_age_p99 > 30s (the latter is the business-pain pager: 'user paid but no confirmation email for 30 min'). Mitigation: rolling restart preserves cluster availability; sustained 5min broker outage triggers add-replica via Cruise Control rebalance.

Worker · Fan-outGo + Kafka consumer group + per-priority pool

Consumes from Kafka topics by priority. P0 (notifications.p0.ticket-delivery): emits email + push notifications via SES / SendGrid + FCM/APNs — the user gets their ticket / booking PDF. P1 (inventory.changed): writes the seat-sold/room-booked flag back to Elasticsearch via bulk indexer so search reflects sold-out within ~1s. P3 (partner.p3.marriott-pms): pushes to hotel PMS / barcode-mint service / CRM. Priority isolation via separate consumer groups + thread pools.

Why it exists. Need to do side effects (email, push, partner sync) AFTER the booking commits, WITHOUT blocking the confirm response. Rejected 'sync fan-out from confirm-svc' because a SES/SendGrid timeout would block the confirm response — user sees 'your card was charged but no confirmation', which is the catastrophic UX failure mode of the dual-write divergence. Async fan-out via Kafka unblocks the confirm and gives each downstream consumer its own retry budget.

When it fails. P3 partner consumer hits a poison message (malformed booking event) and stalls the partition. Detected by consumer.same_offset_for_seconds > 60. Mitigation: bounded retries (3), then route to DLQ; per-partition isolation means a poisoned partition doesn't block siblings; schema validation at the outbox writer side (NOT just consumer) catches most malformed events before they hit the bus.

Service · Fraud ScoringJava + Sift SDK + Stripe Radar fallback

Called synchronously by Confirm Service before the Stripe charge. Sends {user_id, booking_value, ip, device_fingerprint, velocity_signals} to Sift / Stripe Radar; returns a 3-tier verdict (accept / review / block). Block → confirm-svc returns 402 fraud_decline (industry-standard ~3–5% of legitimate confirms hit this path). Review → confirm-svc still charges but flags the booking for ops review. Accept → confirm-svc proceeds to charge.

Why it exists. Need a pre-charge fraud check at 10K confirms/sec — chargebacks have a 60-day settlement window and a ~1–3% chargeback rate at our class of business eats the margin. Rejected 'rely on Stripe Radar alone' because Radar fires AFTER the charge succeeds, and a chargeback unwinds the inventory commit days later — by then the seat / room can't be resold. Rejected 'no fraud scoring' because at the 20:1 bot ratio of an onsale, scalper-friendly volumes leak through. Pre-charge scoring at <200ms is the load-bearing mitigation.

When it fails. Fraud-svc cascade failure during onsale (Sift's published 2023 outage pattern). Detected by fraud_svc.5xx > 0.5% for 60s AND fraud_svc.latency_p99 > 500ms. Mitigation: circuit-breaker opens, confirm-svc proceeds in fail-open mode; ops review queue inherits the burden. Trade-off explicitly accepted: brief widow of higher chargeback risk vs. blocking all confirms during a fraud-svc blip.

Worker · Compensation (Saga)Go + Kafka consumer + Temporal-style state machine

Subscribes to Kafka bookings.failed_post_charge (emitted by confirm-svc when the inv-db SERIALIZABLE txn fails AFTER Stripe charge succeeded). For each event: (1) calls Payment Provider POST /refunds with a refund-scoped idempotency-key (refund:{payment_intent_id}) + charge_id to reverse the charge; (2) writes a compensation_log row in inv-db with {idempotency_key, charge_id, refund_id, status}; (3) emits bookings.refunded for downstream notification. Idempotent via idempotency_key — replays are safe.

Why it exists. Confirm-svc cannot commit booking AND charge in one ACID transaction — payment is an external system. Rejected 'pretend it can never fail post-charge' because a 1-in-100K deadlock at 10K confirms/sec is one stranded charge every 10s during peak. Rejected 'manual ops reconciles every morning' because the user sees their card debited and no booking — a P1 incident-grade UX. Following Airbnb's published saga pattern: the orchestrator drives the workflow; the compensation step is a real worker, not a runbook.

When it fails. Refund itself fails (Stripe rejects, e.g. card closed). Detected by compensation.refund_5xx_rate > 5% for 5m. Mitigation: after 5 attempts route the event to a manual-ops queue; ops issues an ACH refund / store credit out-of-band. The cardinal contract: the user is MADE WHOLE within 24h regardless of automation success.

Service · PSP Webhook ReceiverGo + HMAC verifier + Postgres dedup table

Public HTTPS endpoint for asynchronous events from Stripe / Adyen (charge.succeeded, charge.refunded, charge.dispute.created, payment_intent.canceled). Verifies HMAC signature against per-PSP secret; claims (psp, event_id) in a Postgres dedup table (30-day TTL because PSPs replay for up to 30 days); publishes the canonical psp.event to Kafka; returns 200. Downstream: compensation-svc consumes for refund acks; fanout consumes for the customer-facing 'card charged' email; confirm-svc's cancel path consumes for chargeback-triggered cancellations.

Why it exists. Need a separate ingress for the PSP back-channel that does NOT share fate with our public API gateway. Rejected 'reuse gw + auth bypass' because a misconfigured auth rule on gw blocks PSP webhooks, which then triggers PSP retries (24h × N webhook subscribers) — published Stripe outage retro pattern. Dedicated webhook-svc with PSP-specific HMAC verification isolates the back-channel from public API auth changes.

When it fails. Bad HMAC (the cited Square 2023 cert-bundle incident pattern). Detected by webhook.hmac_failure_rate > 0.1% for 60s. Mitigation: per-PSP HMAC secret rotation runbook with manual T-30/14/7 day alerts; failing-closed (returning 401) is correct — PSP will retry.

Object Store · Audit SinkS3 + Glue catalog + Parquet (Iceberg table format)

Append-only durable log of every booking + payment event for 7-year PCI / financial regulatory retention. A dedicated Fanout-Worker consumer group subscribes to bookings.*, payments.*, and psp.event Kafka topics; batches records into Parquet files (1 file per minute per topic), writes to S3 with Glue/Iceberg schema management. Read path: Athena / Trino for ad-hoc audit queries.

Why it exists. PCI / SOX / state-AG audits demand a tamper-evident booking + payment log retained 7+ years. Rejected 'keep it all in Postgres' because 7 years × 100M bookings/yr × 2KB = ~1.4 TB raw on the OLTP cluster — drags every backup, every replica restore, every schema migration through cold data nobody queries. Rejected 'one row per event in DynamoDB' because Athena cannot meaningfully run aggregate audit queries against unindexed DynamoDB — Parquet on S3 is the published cheap-cold-store pattern (Snowflake / Databricks / Iceberg lineage).

When it fails. S3 regional outage (us-east-1 2017, 2021 incidents). Detected by s3.put_5xx_rate > 1% for 60s. Mitigation: cross-region replication to us-west-2 (lazy, eventual); buffer-and-replay from the Kafka 7-day retention if S3 is down for hours. Audit-sink is NEVER on the user-facing critical path — it can be silent for hours without user impact.