Distributed Unique ID Generator — a worked solution

What each component is for

Service · Feature FlagGo 1.22 + Postgres + Redis edge cache

Internal feature-flag service (LaunchDarkly-like, but homegrown over Postgres + edge cache). The SDK polls every 30 s for two flags: idgen.rpc_enabled (per-region) and idgen.dr_mode (per-region). On dr_mode=us-west-2, the SDK switches RPC fallback target to the DR region; on rpc_enabled=false, the SDK refuses RPC fallback and forces UUIDv7-only mode for the caller. Also the kill-switch surface for the deploy-controller's auto-rollback action.

Why it exists. Two scenarios need a dynamic-config surface that doesn't require code or config-file deploys: (1) a bad SDK release ships and we need to disable RPC fallback platform-wide within seconds (the kill-switch); (2) regional DR step #1 ("freeze callers") needs to flip atomically across all callers without a coordinated rolling restart. We considered baking the flag into the gateway's response headers, rejected because the SDK boundary is where the policy lives — the gateway can't disable a UUIDv7 caller because the gateway never sees that traffic.

When it fails. Flag service down → SDK uses last-known cached flag values; no per-call impact for up to 30 s × N (cache survives across polls until pod restart). Catastrophic outage > 30 min → DR-mode flips can't propagate, runbook is to bypass the flag service via a coordinated SDK config-deploy (slower, but safe). Flag misconfiguration (operator types dr_mode=us-west-3 instead of us-west-2) → callers route to nonexistent region; mitigated by enum-validated flag types and a 1% canary rollout on flag changes.

Client (caller service + SDK)polyglot SDK (Go / Java / Python / Rust)

Internal service that needs an ID for a write. Wraps two strategies in one polyglot SDK: (1) generate a UUIDv7 in-process via RFC 9562 — zero network calls, ~2 µs/ID, the default for ~85% of callers; (2) RPC fallback to the Snowflake service for cross-language services or callers that need a 64-bit BIGINT instead of a 128-bit UUID. The SDK also wraps the Leaf-segment 'allocate-range' API for callers that need short, sequential, human-readable IDs (invoice numbers, support tickets).

Why it exists. Modeled as a node because the SDK is the single load-bearing decision surface — 'do I take a network hop on every write?' The cheapest correct answer is 'no, generate UUIDv7 in-process' (Shopify's payment system reported 50% INSERT improvement vs UUIDv4). We considered making this an external library only and not drawing it on the canonical, but on-call needs to see the SDK as the first hop because the most common ID outage is a bad SDK release shipping with a clock-rewind bug — that's the node that pages.

When it fails. Bad SDK release ships a regression in UUIDv7 monotonicity (per-backend counter forgotten on a clock_gettime branch) → duplicate IDs across processes inside the same ms. Detected by the downstream Uniqueness Audit Worker within ~24 h, by which point ~10 K dups can be in storage. Mitigation: progressive delivery on platform-libs (1% → 10% → 100%) gated on a synthetic write-then-uniqueness probe per canary; SDK includes a self-check that hashes the last 1 M IDs and asserts uniqueness in-process at debug builds.

API GatewayEnvoy 1.30 + SPIRE ext-auth

Internal-only API surface for the ID platform. Terminates mTLS from caller services, validates SPIFFE SVIDs via ext-auth, enforces per-caller rate limits (50 KQps default, override per service), routes /snowflake/* to the Snowflake service and /segment/* to the Segment Allocator. NOT a public surface — there is no internet edge in front of this; ID generation is an internal infrastructure dependency.

Why it exists. Single tenancy / authn enforcement point so the Snowflake and Segment services can stay focused on the ID-issuance hot path. Without a gateway, every backend grows its own SVID validation, its own rate limiter, and its own observability — and we lose the per-caller blast-radius cap. Considered exposing services directly via the mesh sidecar (no central gateway), rejected because cross-service rate limits and audit logging needed a single chokepoint when one caller starts misbehaving.

When it fails. Gateway down → the RPC and segment paths fail (~15% of platform writes) but UUIDv7-in-process callers (~85%) keep working — graceful degradation. Multi-AZ deploy + LB health-checks ensure single-AZ loss is automatic. Whole gateway tier dies → mesh sidecars fail open to a static fallback list of Snowflake-svc IPs (last-resort fast path, no rate limit) for 60 s while gateway recovers; auditable via mesh access logs.

Auth / Identity (SPIRE)SPIRE Server 1.9 (HA on embedded etcd)

Issues short-lived (1 h) SVIDs to every workload in the platform via the SPIRE agent on each node. Gateway's ext-auth filter calls SPIRE Server to validate inbound SVIDs on every request. Also rotates the Snowflake/Segment service certs continuously — no static service-account credentials anywhere in the platform.

Why it exists. ID generation is the most-called internal API in the platform — every write depends on it — so a stolen mTLS cert or shared API key would be the highest-leverage credential in the system. Short-lived SVIDs cap the blast radius of a leaked cert at the rotation window. The alternative (long-lived static service tokens) was rejected because cert-rotation incidents were paging on-call quarterly.

When it fails. SPIRE Server outage → existing SVIDs continue to validate via the agent's cached trust bundle for the SVID TTL (1 h). New workloads (autoscale, deploys) can't get SVIDs and won't pass auth → blocks rolling deploys but doesn't break the steady-state ID hot path. Hard outage > 1 h → all SVIDs expire and the entire platform stops accepting RPCs; on-call action is restore SPIRE from etcd snapshot, no data loss.

Snowflake ID ServiceGo 1.22 gRPC, SPIFFE mTLS, chrony local

Issues 64-bit Snowflake-shape IDs over gRPC: (1 sign | 41 ts-ms | 10 worker | 12 seq) — Twitter 2010 layout, custom epoch 2020-01-01. Each pod holds an etcd-leased worker-id and generates IDs locally with no per-call coordination. Reads CLOCK_MONOTONIC for sequence-delta within a ms; reads CLOCK_REALTIME (chrony-disciplined) only for the high-bit timestamp. Halt-on-rewind: refuses to mint if the wall-clock goes backward by more than 5 ms within a single pod's session.

Why it exists. Some callers need 64-bit IDs — legacy schemas with BIGINT FKs, sortable database cursors, ProtoBuf int64 fields where switching to a 128-bit type is a multi-quarter migration. UUIDv7 in-process is the right default but doesn't help these callers. Considered making them all migrate to UUIDv7, rejected because the migration cost (schema + downstream consumers + analytics queries) is months and the RPC fallback is operationally cheap once the bit-budget is sized. Twitter's published ~10K/sec/process realizes ~30K/sec in modern Go gRPC without GC pauses — we size the fleet on the realized number, not the bit-budget ceiling.

When it fails. Single pod with bad clock (NTP drift > 5 ms) → halt-on-rewind kicks in, pod refuses to mint, gRPC returns UNAVAILABLE → mesh load-balancer routes around it within one health check (~5 s). Whole fleet sees correlated NTP drift (bad upstream stratum-1) → all pods halt-on-rewind together → ID RPCs fail platform-wide, BUT UUIDv7-in-process callers (85%) keep working. On-call signal: idgen_clock_rewind_total > 0 over 1 min (Cloudflare-2017 leap-second class). Bad deploy that breaks the lease-renewal loop → leases expire after 10 min, pods refuse to mint, platform write QPS for legacy callers drops to zero — auto-rollback gated on synthetic write-then-read uniqueness probe.

Segment Allocator (Leaf-segment)Go 1.22, pgx, pre-fetch goroutine

Hands out ID ranges via Meituan Leaf-segment double-buffer pattern: each pod holds an active in-memory segment of step=1000 IDs and a pre-fetched 'next' segment. When the active segment crosses 90% used, a background goroutine UPDATEs the (biz_tag, max_id) row in Postgres and atomically swaps the next-buffer in. Per-call latency is sub-ms (in-memory counter bump + bounds check); a DB hit happens once per 1000 IDs.

Why it exists. Two caller classes need this: (1) human-facing sequential IDs — invoice numbers, order numbers, support ticket IDs — where customers expect a roughly monotonic, short integer; (2) Postgres B-tree FKs that prefer dense bigint over sparse Snowflake IDs (~2× tighter B-tree leaves on dense vs Snowflake-spaced inserts). UUIDv7 doesn't satisfy 'short' or 'sequential'; Snowflake doesn't satisfy 'dense'. Considered a bare DB SERIAL per biz-tag, rejected because the per-call DB roundtrip caps throughput at ~500 IDs/sec/biz-tag (Postgres synchronous_commit=on, 5-15 ms tx) and we need ~50K IDs/sec aggregate.

When it fails. Segment-table primary down → pods serve from active + pre-fetched buffers (~9 min worth of headroom at peak QPS for the typical biz-tag) before refusing further allocates. Post-failover, dual-master odd/even on the segment index (each master advances max_id by 2 × step, owning alternate step-blocks — the Flickr pattern generalized to ranges) means the surviving master keeps issuing without waiting on promotion. Segment burn on pod restart is bounded to step (1000 IDs/biz-tag/restart) — leaves gaps in the ID sequence that downstream analytics must tolerate. Hot biz-tag (one tenant burning segments 100× faster than others) → DB write contention on that one row; mitigated by per-biz-tag rate-limit at the gateway and an alert on segment_fetches_per_biz_tag_per_min.

Lease Manager (Worker-ID Reconciler)Go 1.22, etcd clientv3

Background controller that grants and renews worker-id leases. On Snowflake pod boot, the pod calls the Lease Manager's local sidecar which acquires an EPHEMERAL_SEQUENTIAL slot in etcd (/snowflake/{region}/workers/). The Manager renews leases every 1 min; on session-loss it FAIL-STOPs the pod (forces a restart and re-acquire) rather than silently re-claiming the same slot — Discord's published lesson on dup-IDs from silent re-acquire.

Why it exists. Worker-id assignment is the single highest-leverage piece of state in the system: two pods with the same worker-id silently corrupt the entire ID space. Concentrating the lease lifecycle in a dedicated reconciler (vs each pod talking to etcd directly) lets us enforce the fail-stop policy uniformly, instrument lease churn centrally, and gate the cold-start thundering herd at the SDK boundary. Considered putting this logic in a library inside the Snowflake service, rejected because the policy change ('stop minting if your lease has expired') needs to be enforceable from outside the pod when the pod's clock has rewound.

When it fails. Lease Manager down → existing Snowflake pods continue minting from cached leases (fail-static, up to lease TTL = 10 min). New pods can't boot — autoscale freezes, but steady-state ID hot path keeps working. Manager fleet AND etcd both down → hard fail at lease-TTL expiry; on-call action is restore etcd from snapshot then resume Manager. Bug in the renewal loop (e.g. silently retrying a session-expired lease) is the worst-case — manifests as worker-id collision detected by downstream Uniqueness Audit; runbook is freeze deploys, identify the bad pod via count(distinct host) by (worker_id), fail-stop it, then root-cause.

Lease Cache (Redis)Redis 7 (cluster mode)

Read-through cache for worker-id leases — the Snowflake service consults this on every Nth ID generation (default: every 1024 IDs) to verify its lease is still valid before issuing the next batch. Holds (worker_id → {host, lease_expires_at, dc_id}) keyed by worker-id, plus a reverse (host → worker_id) index for diagnostics. TTL set to lease-expires-at; eviction policy ttl-only so we never serve a logically expired lease.

Why it exists. Without a cache, the verify-lease check would have to hit etcd on every ID call (or every 1024) — adds 5-10 ms of network latency to a sub-ms operation, defeats the point of in-pod ID generation. We considered just caching the lease in pod memory (no shared cache), rejected because the Lease Manager's emergency 'revoke' command needs a place to publish that the pod sees within milliseconds — the cache is the broadcast surface. Hot path = cache hit (~99.9%); cold path = cache miss → etcd fetch → cache write.

When it fails. Cache cold (post-restart, all keys evicted) → first ID call per pod cache-misses, falls through to etcd → 12 pods × 1 lookup = 12 etcd reads, trivially absorbed. Cache cluster fails completely → Snowflake pods fall back to a 1-second pod-local TTL on their cached lease (degraded mode). The dup-window math: 1 s × 30 K IDs/sec/pod = 30 K worst-case silent dup IDs per concurrently-misissued slot, IF (a) an emergency Lease Manager revoke is in flight AND (b) the freed slot is granted to another pod within that 1 s window. The streaming overlap probe is sized to catch this within seconds. Accepted because (1) emergency revokes happen < 1×/quarter, (2) the bounded blast radius is observable rather than open-ended. On-call signal: lease_cache_hit_rate < 0.99 for 5 min → ticket; lease_cache_unavailable_total > 0 for 1 min → page (because of the dup-window math above, not because of latency).

Coordinator (etcd)etcd 3.5 (5-node, 3-AZ)

Source of truth for (1) Snowflake worker-id leases (/snowflake/{region}/workers/{seq} ephemeral keys), (2) Lease Manager leader election, (3) Segment Allocator service leader for the hot biz-tag pre-fetch coordinator. Watches let the Lease Manager and Lease Cache react to lease changes within the etcd --election-timeout budget (default 1 s).

Why it exists. Worker-id leasing requires linearizable coordination — two pods cannot agree to share a slot, ever. Raft gives us linearizable writes with provable safety under network partitions. Considered ZooKeeper (Twitter's original choice) — rejected because etcd's HTTP/2 + gRPC is friendlier to operate, and the SPIFFE story is cleaner. Considered no coordinator at all (worker-id from k8s downward-API: hash of pod-name) — rejected because pod-name collisions across regions would silently allocate the same worker-id, and we'd lose the centralized fail-stop enforcement.

When it fails. Quorum loss (2 of 5 nodes alive) → all writes blocked, watches stale → no new leases granted. Existing Snowflake pods keep minting from cached leases for up to 10 min (lease TTL); steady-state ID hot path unaffected. Manager fail-stops pods whose cached leases expire if it can't renew → progressive write outage for legacy callers, with pace = lease-TTL / (number of pods at lease-expiry concurrence). On-call signal: etcd_server_has_leader == 0 for 30 s → page; etcd_server_proposals_failed_total rising → ticket. Recovery = restore the lost-AZ etcd nodes and let the cluster re-converge.

SQL DB · Segment PrimaryPostgres 16 (dual-master + DR replica)

Stores the (biz_tag, max_id, step, version, updated_at) row per business type that uses the segment allocator. Every segment fetch is UPDATE leaf_segment SET max_id = max_id + step, version = version + 1 WHERE biz_tag = ? RETURNING max_id. Single-row hot, ~50 segment-fetches/sec at peak (50K IDs/sec / step 1000). Also stores the audit_event table (segment-issued events) written in the same transaction as the segment UPDATE — that's the load-bearing outbox.

Why it exists. Centralized source of truth so every Segment Allocator pod sees the same monotonically-increasing max_id per biz-tag. Considered a fully decentralized Sonyflake-style allocation (each pod owns a worker-id and embeds it in the ID), rejected because the human-facing 'short, sequential' requirement is incompatible with worker-id bits in the ID. The dual-master pattern (Flickr odd/even applied to the segment index, so each master advances its cursor by 2 × step and owns alternate step-blocks) survives a single primary loss without write outage at the cost of accepting non-strict monotonicity across the two masters.

When it fails. Primary down → segment fetches fail; Allocator pods serve from buffer (~9 min headroom at peak). Dual-master mode means the surviving master keeps issuing without waiting on Patroni-style promotion. Replication lag spike → DR replica stale; failover would issue dups within the lag window — mitigated by SAFETY_BURN. Hot biz-tag DOSes the row with sub-100 ms UPDATE contention → row-lock waits show in pg_locks. Explicit knobs: gateway per-biz-tag token-bucket = 200 segment-fetches/min default (override per-tag); alert at segment_fetches_per_biz_tag_per_min > 1000 (ticket, SRE-platform owner) and > 5000 (page, on-call SRE-platform). Auto-step-bump (1000 → 10000) fires on > 5000 fetches/min sustained 5 min — flips that biz-tag from 1 fetch/sec to 1 fetch/10s at the cost of larger restart-burn.

SQL DB · Segment DR ReplicaPostgres 16 logical replica

Async logical replica of the Segment Primary, hosted in a different AZ from both masters. Two jobs: (1) DR target — promoted on a regional Primary loss (RTO ~5 min, manual gate because dual-master already covers AZ loss); (2) read-only target for the nightly Uniqueness Audit Worker, which scans the audit_event table for duplicate IDs without putting query load on the production primaries.

Why it exists. Dual-master gives us AZ-loss survival but not a clean snapshot for the audit. The nightly segment range-overlap self-join over segment.allocated rows is heavy (sequential scan over 24 h of issuance events) and would saturate the primaries' buffer cache mid-day (the worker-id lease-overlap check runs against the audit-sink Parquet, not here — lease events never land in Postgres). Rejected: running the audit against the primaries with a low-priority resource group — Postgres doesn't have a robust IO priority story, and the audit's worst-case latency is hours.

When it fails. Replica down → audit job has no read target → uniqueness audit lapses → silent dup IDs could accumulate undetected for days. Detection: audit_job_last_success_age_hours > 25. Mitigation: secondary audit job runs against the surviving primary at 04:00 UTC (degraded mode, with rate-limited reads). DR scenario: replica behind primaries by > 5 min lag triggers a page (RPO breach).

External · NTP / Chrony Poolchrony 4.x + GPS stratum-1

Out-of-process time source. Each Snowflake pod runs chrony as a host daemon syncing to two stratum-1 sources (cellular-grade GPS-disciplined oscillators on bare-metal NTP servers, plus Cloudflare's time.cloudflare.com as backup). The Snowflake process itself reads clock_gettime(CLOCK_REALTIME) (vDSO, ~25 ns) for the high-bit timestamp and CLOCK_MONOTONIC for the intra-ms sequence delta — chrony only adjusts the wall clock, never the monotonic.

Why it exists. Snowflake's correctness invariant is lastTimestamp ≤ now() — a clock rewind breaks it. Bad NTP is the #1 documented cause of distributed-ID dups (Cloudflare 2017 leap-second; Meta migrated from ntpd to chrony to drop drift from 10 ms to 100 µs). We model NTP as an explicit external dependency rather than pretending it's free because the dependency contract — '|drift| ≤ 5 ms over 1 min' — is what halt-on-rewind enforces against. Considered relying on cloud-provider NTP only (AWS Time Sync), rejected because correlated drift across an entire AZ is a real failure mode and a second independent source is cheap insurance.

When it fails. Single NTP source goes bad → chrony picks the other; no impact. Both NTP sources go bad → chrony enters 'unsynchronized' state, drift grows monotonically. Snowflake pods detect via monotonic_vs_wall_drift_ms > 5 and refuse to mint. Leap-second style event (clock jumps backward) → halt-on-rewind triggers within microseconds of the rewind; gRPC returns UNAVAILABLE; mesh routes around the affected pod. Detection: page on idgen_clock_rewind_total > 0 over 30 s. Catastrophic: bad NTP source feeds a leap-second-style smear bug into all pods simultaneously → entire fleet halts; UUIDv7 callers (85% of platform) keep working.

Stream · Audit EventsKafka 3.7 (3 brokers across 3 AZs)

Kafka topic carrying the structural issuance events: snowflake.lease.issued (worker-id grants, renewals, revokes) and segment.allocated (each Leaf-segment range [from_id, to_id]). It deliberately does NOT carry per-ID events — ~13 B individual IDs/day would be prohibitive — so a duplicate is detected structurally: two overlapping segment ranges, or two overlapping worker-id leases. Consumers are (a) the Audit Sink that archives to S3, (b) the Uniqueness Audit Worker. Producers are the Snowflake and Segment services; for the Segment side this is the outbox half of the segment-allocation transaction.

Why it exists. Without a centralized audit log, after-the-fact 'why did we issue ID X' forensics is impossible — you'd have to scrape logs across 18 pods plus the segment-table version history. The audit stream is also the substrate for the uniqueness probe: the consumer can spot dups in an issuance window without querying every downstream consumer table. Considered writing audit rows to Postgres directly (synchronous), rejected because the per-call latency budget for ID issuance is < 1 ms and we cannot afford a synchronous fanout to a separate store.

When it fails. Kafka cluster down → audit producers buffer in pod-local disk-backed outbox (60 s capacity); under prolonged outage, ID-issuance still works (we don't block on audit) but the audit log gets a gap. Detection: audit_outbox_lag_seconds > 60 → page. Hot partition (one biz-tag dominates) → consumer lag grows on that partition; mitigation is partition reassignment via the partition-key hash including a sub-key. Poison message → DLQ topic + alert on first DLQ message.

Worker · Audit Outbox DrainerGo 1.22 + pgx + Sarama Kafka producer

Background worker that polls the audit_event table on the Segment Primary every 200 ms for rows with published=false, publishes them to the audit-stream Kafka topic, and on Kafka-publish-success marks the row published=true (it does NOT delete it — audit_event doubles as the retained audit trail the nightly uniqueness scan reads over a 25 h window; space is reclaimed by dropping day-partitions after 90 d, not per-row). Idempotent on the Kafka side (enable.idempotence=true + event_id as the dedup key on consumers). This worker — not the Segment Allocator — is what makes the segment+audit outbox actually drain to Kafka.

Why it exists. The segment UPDATE and the audit_event INSERT are written in the SAME Postgres transaction (single-store outbox); the actual Kafka publish has to happen out-of-band, otherwise the segment service would block on Kafka availability and we'd lose the at-least-once delivery contract. Considered making the segment service itself drain its own outbox in a goroutine, rejected because (a) the goroutine's failure modes are invisible to on-call (no separate alert surface), (b) restart of the segment service would orphan the in-flight publishes. A dedicated drainer with its own SLI (audit_outbox_drainer_lag_seconds) and its own deploy unit is operationally correct.

When it fails. Drainer down → audit_event rows accumulate in Postgres; segment-allocator hot path unaffected (it doesn't depend on the drainer). Detection: audit_outbox_drainer_lag_seconds > 60 ticket, > 600 (10 min) page. Worst case: drainer fleet down for hours → audit_event table grows; mitigation is the table is partitioned by issued_at-day, capacity 100 M rows/day fits comfortably. Drainer publishes a row but crashes before marking it published → Kafka has the event, Postgres still shows it unpublished, next poll re-publishes → at-least-once delivery; consumers dedup on event_id.

Object Store · Audit ArchiveS3 + Glacier IR + Kafka Connect S3 sink

Long-term cold storage of every ID-issuance audit event. A Kafka Connect S3 sink consumer batches events from audit-stream into 5-minute Parquet files partitioned by (date, svc). Holds the full historical record of which worker_id was issued to which host at which lease_start — the substrate for any after-the-fact compliance or debugging query.

Why it exists. Kafka retention is 30 days; audits, security investigations, and data-lineage queries can require years of retrospect (SOX 7 years for financial systems, GDPR-defensible retention windows for compliance). S3 + Glacier is the cheapest durable substrate. Considered a wide-column DB (Cassandra) for queryable audit history, rejected because we have no real-time query pattern on years-old audit data — the queries are batch-analytics in a warehouse, and Parquet-on-S3 is the warehouse-native format.

When it fails. S3 PUT errors → Kafka Connect retries with exp-backoff, holds offset commit until success; under prolonged S3 outage, Kafka retention (30 d) is the buffer. Sink fails to keep up → consumer lag grows; alert on s3_sink_lag_minutes > 30. Worst case: silent corruption of a Parquet file — mitigated by daily checksum job that re-reads a sample of files and verifies against the source Kafka offsets.

Worker · Uniqueness AuditGo 1.22, in-memory interval index (RoaringBitmap for range sets)

Two pipelines in one worker fleet, both structural (the stream carries ranges + leases, not per-ID rows). (1) Streaming probe — consumes audit-stream and maintains, per biz_tag, an interval index of allocated [from_id, to_id] ranges; any new segment.allocated that overlaps an existing range is a dup candidate, and for snowflake.lease.issued any two live leases for the same worker_id with overlapping validity are a dup candidate. Overlap detection is exact, so a hit is a confirmed dup — the worker pulls the two conflicting records from the audit-sink Parquet for the on-call. (2) Nightly batch probe — at 03:00 UTC over the last 25 h: a range-overlap self-join on segment.allocated against the Segment DR Replica (a.biz_tag = b.biz_tag AND a.from_id <= b.to_id AND b.from_id <= a.to_id AND a.event_id <> b.event_id — segment rows are txn-guaranteed in Postgres), plus a worker-id lease-overlap pass over snowflake.lease.issued read from the audit-sink Parquet (lease events are published straight to Kafka and archived to Parquet — they are never written to Postgres, so the lease check cannot run against the DR replica). Any overlap fires a P0 page.

Why it exists. ID-uniqueness is the platform's single hardest-to-test invariant — it can be silently wrong for hours before any caller notices, and by then dups are durable in storage across many tenant systems. The downstream uniqueness probe is the load-bearing SLI; without it, the failure modes that produce silent dup IDs (worker-id collision, clock rewind on one pod, segment-failover-burn) are undetectable. Considered making each consumer system run its own dup-check, rejected because that's N teams owning the same control and they will skip it; a centralized probe per audit-event source is correct and observable.

When it fails. Streaming probe consumer lags → window grows beyond 24 h → an overlap against a range/lease already evicted from the in-memory window could be missed by the streaming path (the nightly scan + the audit-sink Parquet cross-check backstop it). Detection: audit_consumer_lag_seconds > 300 page. On restart the interval index is rebuilt from the audit-sink. Worst: the worker silently fails AND the nightly scan also lapses → silent dup window of days; mitigated by audit_job_last_success_age_hours > 25 paging at 25 h regardless of cause.

External · KMS (audit + SVID keys)AWS KMS (per-region, HSM-backed)

AWS KMS (or equivalent HSM-backed key store) holding two key categories: (1) the SSE-KMS data key for the audit-sink S3 bucket — every Parquet write does an AWS:Encrypt call to wrap a per-object data key; (2) the intermediate CA signing key the SPIRE Server uses to issue SVIDs. KMS calls are out of the per-call ID hot path — only audit writers and the SPIRE Server hit KMS, and SPIRE caches signed material aggressively.

Why it exists. Modeled explicitly because KMS outage is a real correlated-failure mode: it disables both audit immutability (S3 sink can't encrypt new objects) and SVID rotation (existing SVIDs survive their TTL but no new ones can be minted). On-call must see the KMS dependency on the canonical to understand why a regional KMS incident pages the ID platform — without this node, KMS shows up as a phantom dependency in post-mortems.

When it fails. KMS regional outage → S3 audit-sink writes fail with KMS:Encrypt error → Kafka Connect S3 sink retries with exp-backoff and pauses offset commits → consumer lag grows on audit-stream. Detection: s3_sink_kms_error_rate_per_min > 0 for 5 min → page. SPIRE Server: existing SVIDs (1 h TTL) keep validating, but no new workloads can boot — autoscale freezes. KMS keypair rotation incident (unlikely but real): old key disabled before all consumers can re-key → audit replay can't decrypt historical Parquet; mitigated by a 30 d key-disable grace window before deletion.

Monitoring (Prometheus + Alertmanager)Prometheus 2.50 + Alertmanager

Scrapes metrics from every node in the platform every 15 s; evaluates burn-rate alerts on the four load-bearing SLIs: (1) ID-issuance availability (99.999% SLO), (2) duplicate-ID rate (target = 0; alert on first occurrence), (3) lease-renewal success (any failure = ticket; sustained = page), (4) NTP sync (chrony fell out of sync = page within 5 min). Pages via PagerDuty; tickets via the standard issue tracker.

Why it exists. ID generation is the platform's single highest-leverage dependency — a 99.9% ID service degrades the entire platform to 99.9%; we need explicit observability to claim 99.999%. Without burn-rate alerts on the four SLIs, on-call learns about ID outages from downstream consumer pages (which take longer to fire and have wider blast radius). Considered using cloud-provider managed monitoring (CloudWatch), rejected because the dup-ID burn-rate alert needs custom recording rules and per-SLO budget math that we want explicitly authored.

When it fails. Prometheus down → alerts don't fire; the platform keeps working but on-call is blind. Mitigated by a heartbeat alert (Prometheus pushes a 'I am alive' metric to a dead-man's-switch external service every 30 s; on absence > 2 min, the external service pages). Alertmanager misroute → a dup-ID page goes to the wrong on-call; mitigated by routing-config-as-code with PR review.