Collaborative Editor (Google Docs) — a worked solution

Typical clarifications to surface:

• Rich text or plain text? Affects op model (ProseMirror steps vs Yjs Y.Text). Assume rich text — paragraphs, formatting, embedded objects.
• Offline editing required? Yes — laptop-sleep-then-wake within hours; reject pure-server-required designs.
• Multi-device same user? Yes — laptop + phone editing concurrently is normal.
• Scope note. This canonical is the real-time editing core. Comments, suggestions (track-changes), and @mention notifications are intentionally out of scope — they layer over the same oplog spine (add an op-kind discriminator; add a consumer group) without changing the write path.
• Multi-region edits to the same doc? Yes for read; for write, home-region per doc (not active-active per op).
• Latency target? p99 keystroke→broadcast intra-region < 200ms (the shared-cursor "feels live" bar). Cross-region eats the WAN RTT.
• Retention? Operational oplog 30d hot, cold tier 7y for compliance. Snapshots versioned forever per workspace policy.
• Authentication? SSO (OIDC/SAML), per-doc ACL, capability tokens with ≤30s lifetime so revocation actually takes effect mid-session.
• Max doc size / max editors? ~50MB / ~50 concurrent writers per doc. Above 50 writers → broadcast-fanout mode.

Assumptions to state (Mid tier, 50M DAU):

• Concurrent WS at peak: 3.75M (7.5% of DAU).
• Active editors at peak: ~500K (13% of online).
• Sustained ops/s system-wide: 2M → peak 8M ops/s.
• Peak deliveries/s (fan-out): 12M/s (mean 1.5 editors/active-doc).
• 30d hot oplog after RF=3 + LZ4: ~1 PB on disk (≈330 TB compressed pre-replication).
• Snapshot every 1000 ops or 60s.
• 5-region active-standby with home-region pinning per doc.

• Open a doc → see current contents (p99 < 300ms warm, < 1.5s cold).
• Type → see your own keystrokes locally instantly (optimistic), see peers' keystrokes within p99 200ms intra-region.
• Multi-cursor / multi-selection visible per editor (presence channel).
• Version history — restore any point in the last 7y (a native side-effect of snapshots + oplog retention; not a separate service).
• Offline edit → automatic merge on reconnect (Figma-style: re-snapshot + replay buffered).
• Permission revoke → mid-session WS close within ≤30s (one capability-token TTL).

• Availability: 99.95% on the read+open path; 99.9% on the write path (writes can briefly degrade during a doc-actor handoff).
• Latency: p99 keystroke→broadcast 200ms intra-region, 400ms cross-region. p99 doc-open 300ms warm / 1.5s cold. p99 reconnect-resume 1.5s.
• Durability: Zero loss of any op the user observed on a second device. RPO=0 on the durable boundary (Kafka acks=all, MIN.ISR=2). Cross-region DR: RPO ≤ 120s on doc content (MirrorMaker-2 oplog replication, alerted at 120s lag); snapshots ride S3 CRR at ~15min p99 with the gap filled by oplog replay; doc-meta ≤ 30s via its async observer.
• Consistency: Strong order per-doc (single-writer doc-actor); causal across docs is sufficient. Read-your-write within a doc is guaranteed by single-writer; across regions it is region-local.
• Conflict-freedom: Real, not aspirational. The doc-actor is the linearizer; OT/CRDT structures used inside it are an implementation choice, not a multi-master substitute.
• Scalability: Horizontal on every tier except the per-doc actor (which is, by design, a single writer). Doc-server pool scales by adding pods; doc-coord scales by sharding the etcd keyspace if a 5-node cluster bottlenecks.
• Security: Per-doc capability tokens with ≤30s TTL; ACL-version-bump invalidates in-flight caps; mTLS service-to-service; KMS-encrypted snapshots; doc-server runs in workspace-scoped tenancy boundaries.

Walk-through (Mid tier, 50M DAU; Workspace tier 4×):

1. Concurrent WS at peak. 50M DAU × 7.5% online + doc-open = 3.75M concurrent WS. Anchored against Slack's published 5M+ peak WS at smaller-than-Workspace scale.
2. Active editors. 13% of online actively typing → ~500K active editors. The remainder are viewing or idle.
3. Sustained ops/s. 500K × 4 ops/s/editor (after 250ms client-side coalesce of fast typing) = 2M ops/s sustained. Google's published 60K–200K coarser-grained ops/s for ~100M DAU lands in the same regime.
4. Peak ops/s. Workday peak multiplier 4× → 8M ops/s peak.
5. Deliveries/s. Mean 1.5 editors/active-doc → fan-out factor 0.5 above the producer → 12M deliveries/s peak. Discord pushes 26M WS events/s on 400 Erlang hosts — same order of magnitude.
6. Doc-server pod count. Each pod holds ~250K hot docs (32GB heap, ~110KB live state per doc — ops since last snapshot + presence + WS refs). At 333K active docs (= 500K editors / 1.5 mean), 2 pods could fit it — but the math that matters is handoff blast radius. Six pods per region × 5 regions = 30 pods caps each pod's blast radius at ~80K open docs per crash (3.75M WS / 1.5 per doc = 2.5M materialized actors ÷ 30), restorable within RTO 5s.
7. ws-gw pod count. 100K active WS / pod (Phoenix bench 2.3M idle, 5× drop active, halved for safety). 3.75M / 100K = 38 pods raw → 12/region × 5 = 60 pods with 1.5× headroom for region failover.
8. Oplog partition count. 8M ops/s peak / max 4K ops/s/partition = 2K → 2048 partitions, RF=3, distributed over 60 brokers. Each broker handles ~33 partition leaders + followers — well within a c6i.4xlarge's 16-core / 4Gbps NIC budget.
9. Oplog hot bytes. 2M ops/s × 200B × 86400 × 30d = 1 PB raw → ÷3 LZ4 → ~330 TB raw, × RF=3 = ~1 PB on disk per region. KIP-405 tiered storage offloads anything older than 24h to S3, so on-broker is ~30 TB/region hot.
10. doc-meta size. 1B docs × 1KB metadata + 100M ACL grants × 200B = ~1 TB. 32 shards × 30GB each. Sized for ACL read/write QPS: ~50K reads/s (capability mint), ~5K writes/s (perm changes).
11. Snapshot churn. 500K active × 1 snapshot/min × 50KB median = 36 TB/day of S3 PUTs. Request charges dominate storage: ~720M PUTs/day ≈ $108K/month at $0.005 per 1K PUTs, vs ~$23K/month for the ~1 PB steady state under the 30d checkpoint GC. The cost lever is PUT count, not storage class — a median doc at 4 ops/s hits the 60s timer with only ~240 ops, so stretching slow-churn docs toward the 1000-op bound cuts PUTs ~4× with the same replay ceiling. Intelligent-Tiering is a non-answer here (50KB median objects sit below its 128KB tiering floor).
12. Cache working set. Two distinct sizing axes: (a) hot-doc lookup cache 1B docs × 2% hot × 50KB = ~1 TB would fit in 10 pods, but (b) presence absorbs 12M events/s peak (presence XADD is the dominant cost), so we run 128 pods to keep per-shard write CPU and AOF fsync queue healthy. Both prefixes co-reside.
13. Capability-token mint QPS. Every concurrent session holds a live capability token — viewers included — so mint load tracks WS count, not typing: 3.75M sessions × 1 mint per 30s TTL ≈ ~125K mints/s steady. auth runs 60 pods → ~2.1K mints/s/pod against an HSM-backed signer's budget (hardware-keyed Ed25519 ≈ 5K signs/s/pod) — headroom for reconnect-storm re-mint spikes.
14. Lease coordinator load. 250K active docs renewed every 5s with batched renewals (one Lease.KeepAliveOnce RPC per pod carries up to 10K leases) = 30 pods × 1 batched RPC / 5s = 6 RPCs/s total — three orders of magnitude under etcd's documented ~10K writes/s ceiling. Without batching, naïve per-doc renewal would be ~50K writes/s and require an etcd shard, which is why batching is non-negotiable.
15. Cross-region. Each doc home-pins. ~20% of editors collaborate cross-region → ~100K editors paying WAN RTT (~80–150ms transatlantic), eating into the keystroke→broadcast budget. SLO has a separate cross-region p99 of 400ms.

Three scale-tier flips (the design must be ready for):

• Single doc-process saturates at ~50 editors / ~500 ops/s for one doc. GC + per-doc serialization queue dominates. Above this → broadcast-fanout mode (writers on actor, viewers on edge-cached snapshot stream).
• Don't shard a single doc. Above 100 concurrent editors, demote viewers to read-only or section-partition; never split write-authority (re-introduces OT divergence).
• Presence flips to compacted Kafka above 10K events/s/doc so cursor noise doesn't poison the inline op path.

POST /api/v1/docs
Content-Type: application/json
Authorization: Bearer <token>

{ "workspaceId": "ws_abc", "title": "Q4 plan", "type": "doc" }

201 Created
{ "docId": "doc_01H...", "homeRegion": "us-east-1", "snapshotUrl": "https://..." }

GET wss://collab.arc.ly/v1/collab/{docId}?lastAckedSeq={seq}
Authorization: Bearer <capability-token, 30s>

# Server upgrades to WS. First server frame:
{ "type": "snapshot", "seq": 12345, "snapshotUrl": "https://...", "epoch": 8 }
{ "type": "ops",      "seq": 12346..N, "ops": [...] }   # delta if lastAckedSeq known

# Client → server (continuous):
{ "type": "op",       "clientOpId": "uuidv7", "parentSeq": N, "op": {...}, "epoch": 8 }

# Server → client (continuous):
{ "type": "op-ack",   "clientOpId": "...", "seq": N+1 }
{ "type": "op",       "seq": M, "op": {...} }            # peer ops
{ "type": "presence", "userId": "...", "cursor": {...} } # ephemeral, never persisted
{ "type": "reconnect-after", "ms": 3000, "reason": "drain" }
{ "type": "access-revoked" }                              # forces clean WS close

GET /api/v1/docs/{docId}/history?cursor=&limit=100&from=&to=
# cursor: opaque, descending snapshotSeq; limit caps at 200; from/to: optional ts range ("versions around date X")
200 { "items": [ { "snapshotSeq": ..., "ts": ..., "by": ... }, ... ], "nextCursor": "..." }   # nextCursor null at end

Op record (stored in Kafka oplog as one record per op, partition key (workspace_id, doc_id)):

Doc-meta (Postgres, sharded by workspace_id):

Snapshot (S3 versioned):

• s3://docs-{region}/ws={workspace}/{doc_id}/{snapshot_seq}.bin — binary-encoded Yrs state, zstd, KMS-encrypted.
• Latest pointer in docs.current_snapshot_seq. Versioned bucket retains every snapshot for 7y per policy.

Why Kafka (KIP-405 tiered) for the canonical oplog and not Postgres? Per-partition strict ordering, a transactional producer keyed by a per-doc transactional.id whose epoch bump fences a stale owner (the WAL-before-ack contract), 30d hot retention with seamless cold tier, and compaction support if we later want a per-key snapshot view. Postgres would need partitioning logic + outbox tables + a separate truncate/cold-store pipeline that we already get for free here.

Architecture summary (request-flow, left to right):

1. Client holds the local Y-CRDT mirror, applies ops optimistically, queues them to IndexedDB, and ships them over a WS to the closest edge.
2. Edge L4 LB + WAF (Katran/Envoy) anycasts traffic to the nearest region; rate-limits, blocks the obvious nasties, terminates only at the gateway.
3. Collab WS Gateway terminates TLS, consistent-hashes the doc to a doc-server pod via doc-coord, forwards op frames in either direction, and on drain emits reconnect-after: jitter(60–600s) to prevent storms.
4. Control API Gateway carries doc CRUD, share/unshare, version-history reads, and mints the presigned URL for the paste-50MB-import path — the blob itself goes client → S3 directly, never through the app tier.
5. Identity + ACL mints per-doc capability tokens with ≤30s TTL; ACL bump on the doc-meta forces token re-mint and pushes access-revoked on existing sockets.
6. Doc-Actor Lease Coordinator (etcd Raft) owns (doc_id) → (owner_pod, lease_epoch). It is the only thing that can promote a new owner; the lease epoch becomes the fencing token on every oplog append.
7. Doc-Actor Pool (Rust + Yrs) holds the in-memory doc, transforms ops via the embedded CRDT engine, appends transformed ops to Kafka with acks=all and the lease epoch, and only then broadcasts to peer subscribers. This is the load-bearing claim of the design.
8. Kafka Oplog (KRaft + KIP-405) is the source of truth — strict per-partition ordering, transactional producer with per-doc transactional.id epoch fencing, 30d hot + 7y cold tiered.
9. Snap-store (S3) holds checkpoints every 1000 ops or 60s, versioned, KMS-encrypted; cross-region async replicated for DR.
10. Doc-meta (Postgres-HA + Citus) holds the durable rows for ACL, version pointer, DLP quarantine flags; sync-replicated so failover RPO=0.
11. Presence + Idem Cache (Redis Cluster) is split into three key-prefix concerns: ephemeral presence (TTL 30s, never persisted), idempotency dedup window (7d, AOF-fsync), and lease lookup cache (5s).
12. DLP / Audit worker is the one derived pipeline included in this core canonical: a separate consumer group on the oplog runs per-tenant policy + writes an immutable audit log to S3 Object Lock; the canonical oplog is never mutated. (Comment/mention notif-workers are the same shape — omitted here to keep the diagram focused on the real-time editing story.)

Data flow on keystroke (the contract path) — two users editing the same doc:

1. User A's ingress: client → edge-lb → ws-gw → doc-server. ws-gw consistent-hashes by docId, so both users land on the same doc-server pod.
2. Single-writer transform: doc-server transforms User A's op against the in-memory Y-CRDT state.
3. WAL-before-ack (durability boundary): doc-server BLOCKS on a Kafka acks=all + min.insync.replicas=2 append, fenced by the lease epoch.
4. Broadcast back: doc-server pushes the transformed op down its bidi gRPC stream to ws-gw, which fans out the WS push to every other subscriber for that doc — including User B (peer-client).
5. User B sees User A's keystroke within the 200ms intra-region budget. Symmetric: if User B also types, their ops enter via peer-client → edge-lb → ws-gw → doc-server and land on the same actor, where the per-doc serialization queue interleaves A's and B's ops in causal order.

This is the moment of collaboration. The single-writer doc-actor is what makes it conflict-free in the strong sense — A's and B's concurrent ops never race against each other on different machines, never need a multi-master merge, never produce divergent server states. The diagram makes this visible: both client and peer-client fan into the same ws-gw → doc-server, and the broadcast leg (doc-server → ws-gw → peer-client) closes the loop.

Data flow on reconnect: client (last_acked_seq=V) → ws-gw → doc-coord (find owner) → doc-server (load snapshot if cold; replay oplog from V → head; push delta or full reload if V < oplog floor) → resume.

Data flow on doc-actor handoff (failover): detected crash (kubelet death, ws-gw's severed gRPC streams) → immediate doc-coord.LeaseRevoke, handoff in ~5s; silent partition → doc-server-A misses renewals (every 5s) and doc-coord expires the lease at the 15s TTL. Either way: doc-server-B acquires new lease (epoch+1) → calls initTransactions() on the per-doc transactional.id (bumping the broker producer epoch) → loads snapshot from S3 → replays oplog tail → starts accepting ops. doc-server-A's now-stale appends are fenced by Kafka (ProducerFenced/INVALID_PRODUCER_EPOCH); A self-evicts on the rejection.

1. Why server-authoritative single-writer beats both pure OT and pure CRDT here.

Pure OT (Google Wave, Mobwrite legacy): Ops travel in any direction; every node must transform every op against every other concurrent op. Subtly buggy (TP1/TP2 properties) — Wave shipped with documented divergence cases. The fix that took Atlassian and Google a decade was the same one we're starting with: a central server that linearizes ops and broadcasts the transformed result, so clients only ever apply server-blessed ops. We don't even call it OT internally — we use Yrs (Y-CRDT) as our state structure because the math is cleaner — but the authority model is OT-style central server.

Pure CRDT (Yjs P2P, Automerge merge-anywhere): Wins for genuinely offline-first / P2P scenarios (think local-first apps, no cloud). Costs: tombstone bloat (Yjs #321, Automerge #337 — every delete is a permanent record until causal-stable GC), vector-clock per op, and on multi-region active-active write the merge surface is real and surprising for users. We get none of CRDT's advantages — we have a cloud, we have a single linearizer per doc — and would pay all of the costs.

Server-authoritative single-writer (this design): The doc-actor is the linearizer. We use Yrs for the data structure because deletes-as-tombstones-with-GC are mathematically convenient, but only one process at a time mutates a given doc, and the lease+fencing-token machinery makes "only one process" a hard guarantee under partial failure, not a hopeful one. Conflict-freedom is real, not approximated.

2. WAL-before-ack and the durability budget.

The doc-actor blocks on Kafka's acks=all (with min.insync.replicas=2) BEFORE acking the client. This is non-negotiable. Consider the alternative: actor broadcasts to peers, then crashes before persisting. Other clients' UIs already show the op. On recovery, the canonical state is older — the user sees their text un-write itself on the peer's screen. This is the worst possible failure for a collab editor, far worse than a 30ms latency hit. So we eat the latency: typical Kafka acks=all in same AZ is ~5ms, cross-AZ ~15ms, comfortably inside the 200ms keystroke→broadcast budget.

3. The fencing-token flow that makes single-writer real.

doc-coord (etcd) issues (owner_pod, lease_epoch). doc-server-A holds lease epoch=8 for doc-X. Network blip — A misses 3 renewals; lease expires. doc-coord promotes B with epoch=9. Meanwhile, A still has WS connections and queued ops; A tries to append to Kafka. Here the mechanism matters: plain enable.idempotence=true does NOT fence a zombie — each producer session gets its own broker-assigned PID, and idempotence only dedups retries within one session by (PID, partition, seq); two different processes get two different PIDs and both would be allowed to write. The real fencing comes from a transactional producer keyed by a deterministic per-doc transactional.id (e.g. oplog-<doc_id>). When B is promoted it calls initTransactions() for that same transactional.id, which bumps the broker-side producer epoch and fences the previous holder — A's next append fails with ProducerFenced (surfaced as INVALID_PRODUCER_EPOCH). So the Raft lease authorizes B to take over; the shared transactional.id + epoch bump is what makes the broker actually reject A. A sees the rejection, self-evicts, drops its WS connections (clients reconnect to ws-gw, get re-routed to B). No double-apply. No ops lost (A's ops never committed, so they never existed in the canonical store; clients re-send via the IndexedDB queue).

The harder partition (doc-server can reach doc-coord but NOT Kafka). A's lease renewals succeed via e13 (different network path); but e14 (Kafka) is unreachable. Without WAL-before-ack, A would broadcast to peers and silently lose ops on recovery. With WAL-before-ack, A simply cannot ack the client — the user sees saving… and types into a buffer. A's safety mechanism: if e14 fails sustained > 5s, A proactively calls doc-coord.LeaseRevoke and drops all WS connections. Clients reconnect, ws-gw re-routes them to a pod with healthy Kafka connectivity, and the IndexedDB-queued ops replay against the new actor. The client UX during the gap is saving… followed by a brief reconnect — equivalent to a 5s flap, no data loss.

Why the oplog is the source of truth, not doc-meta. Earlier drafts of this design coupled the oplog append (e14) and a doc-meta cursor update (e16) into a transactional pair. We rejected that: under actor crash between the two writes, the cursor lags the oplog head and a senior reviewer caught it as a non-outbox masquerading as an outbox. Final design: current_oplog_seq is updated on snapshot, not per-op, via compare-and-set (UPDATE docs SET current_oplog_seq=$new WHERE doc_id=$1 AND current_oplog_seq < $new). The cursor is a hint that bounds cold-open replay; if it lags, replay starts a few hundred ops earlier and Yrs idempotently re-applies them. The oplog's per-partition ordering + transactional-producer epoch fencing is the only durable boundary — there is no second store to coordinate with.

4. Reconnect = re-snapshot + replay (the Figma move).

Old design: client tries to splice missed ops into local state, applying transforms client-side. Subtly buggy class — every causality-violation bug is here. New design: on any reconnect, client throws away local view, downloads fresh snapshot via signed URL through CDN, re-applies its IndexedDB-queued ops on top. p99 reconnect-resume budget 1.5s — well within human "I just refocused this tab" tolerance. Removes a whole class of bugs.

5. Hot-doc fanout (the all-hands doc problem).

A shared all-hands doc with 5,000 concurrent viewers (one author actually typing) saturates a single doc-actor. Fix: above concurrent_clients_per_doc > 200 OR ops_per_sec_per_doc > 500, the doc enters fanout mode. Writers stay on the actor. Viewers get demoted to a compacted Kafka topic + edge-cached snapshot — they see updates within snapshot cadence (60s) plus oplog tail. This is the Google Docs "view-only mode for >100 collaborators" rule and the Figma broadcast pattern, formalized.

6. Snapshot/oplog retention floor.

Oplog retention must satisfy oplog_min_seq < snapshot_seq AND oplog_min_age > max(reconnect_window, ops_replay_for_audit). We pick 30d hot because (a) reconnect window: laptop sleep up to 7d is realistic, doubled for jitter; (b) audit replays: forensic reads up to 30d. Below the floor, the client gets STALE_CLIENT + force reload. We never truncate ahead of snapshot fsync — the silent-corruption path.

7. Multi-region (home-region per doc).

Each doc is created in a home region (workspace's primary region + locality hint). All writes for that doc serialize there. Cross-region collaborators connect locally for low-latency control + presence, but op frames forward to home; their keystroke→broadcast eats one WAN RTT. Snap-store replicates async cross-region (~15min RPO) so DR + version-history reads are local. Doc-meta has a cross-region async observer per shard — read your workspace's ACL locally without a WAN hop. We are deliberately CP per doc; AP on multi-master CRDT was rejected because (a) the single-writer model already guarantees conflict-freedom, (b) no UX team wants to explain "your colleague's edits merged with yours in a way you didn't expect" to enterprise customers.

Trade-offs we accepted (and the alternatives we rejected):

• WAL-before-ack adds ~10-15ms to every keystroke but buys RPO=0 on the durable boundary. Rejected: ack-before-persist (broadcast-then-Kafka). Failure mode of the rejected variant — clients see ops the server later loses — is unacceptable for an editor.
• Single-writer doc-actor means a hot doc has a single-pod ceiling. Rejected: shard the doc by paragraph or section. Section-shard reintroduces cross-shard transform / OT divergence — exactly the bug class we're trying to extinguish. Hot-doc fanout (writer/viewer split) handles the real-world hot-doc shape (many viewers, few writers) without breaking authority.
• Home-region per doc, not active-active per op. Rejected: multi-leader CRDT replication. Costs (tombstone bloat, surprising merges) > benefits (cross-region write latency reduction for the 20% of cross-region collabs).
• Y-CRDT (Yrs) inside the actor instead of OT. Rejected: ProseMirror-style step OT. Yrs is faster (josephg's 56ms vs 5min Automerge benchmark in the Rust port), tombstone GC is well-understood, and we get a binary update format we can persist directly. We don't get CRDT's P2P advantage — but we don't need it.
• etcd for the lease coordinator, not ZooKeeper. Cleaner Raft semantics, simpler client lib, mature in production. Spanner/CockroachDB rejected because we need simple per-key leases, not distributed transactions.
• Capability tokens with 30s TTL (not 24h JWTs). Costs: more frequent token-mint RPS. Buys: revocation actually takes effect mid-session. The auth tier is sized to absorb the mint rate.
• Peripheral product features excluded from this canonical (comments, suggestions/track-changes, @mention notifications). Costs: real-Docs is bigger. Buys: the diagram teaches the load-bearing story — WAL-before-ack + single-writer + lease-fenced Kafka epoch — without dilution. Each excluded feature layers over the SAME oplog with a new op_kind and a new consumer group; the write path never changes. The dlp-scanner in this canonical is deliberately kept as the reference example of that "add a consumer group" pattern.

Open questions for human review:

• Is 30d hot oplog sustainable cost-wise at Workspace tier (4× Mid)? KIP-405 tiered should make this fine, but we should validate broker IOPS at peak rebuild scenarios.
• Should presence flip to a separate Kafka cluster from the start, rather than co-residing in Redis? If presence events spike to 50M/s in pathological viral-doc events, Redis Streams memory pressure is a worry. We've sketched the flip but not load-tested it.
• The 50-editor-per-doc cliff is theoretical. Real-world load tests need to validate it (and the broadcast-fanout transition logic) on production-class hardware.
• Layering peripheral product features (comments / suggestions / @mention push notifications) back on top of this canonical is a one-node-per-feature exercise: add a new consumer group on the oplog (same shape as dlp-scanner), route to doc-meta and/or an external push tier, size the consumer lag SLO independently. Deliberately out of scope here to keep the diagram focused on the real-time editing spine.
• doc-coord scaling beyond 5-node etcd: at ~10M lease ops/s, etcd's write throughput becomes the ceiling. Sketch: shard the etcd keyspace by hash(doc_id) % N across N etcd clusters. Not needed at Mid tier; queue for a 10× re-architecture review.