AI Agent Platform — a worked solution

What each component is for

Client

Three concrete callers share this entry point: (a) an IDE / desktop agent (Cursor, Claude Code, Devin web UI) sending POST /v1/runs, (b) a CLI tool subscribing to GET /v1/runs/:id/events for a long-lived SSE token stream, and (c) a programmatic client (CI, batch job) polling for terminal status. Receives webhook callbacks at a registered URL when a run finishes asynchronously.

Why it exists. Considered modeling each surface as a separate node; rejected because all three share the same auth, idempotency, and SSE-resume contracts. Splitting only matters at the SDK layer (different language SDKs, retry helpers); the system-level failure modes are identical.

When it fails. Naïve clients reconnect SSE without Last-Event-ID and re-render duplicated tokens — looks like a model regression to users. Detection: distinct event_ids/run p95 > 1.05 alerts SDK eng. OpenAI Assistants API has the documented 'stream interruption' bug in this shape (community.openai.com).

CDN · SSE EdgeCloudflare Workers + WAF

Terminates TLS, runs the WAF ruleset (OWASP Top-10 + agent-platform rules: rejects Authorization-header replay attempts, fingerprints scraper UAs), absorbs SSE token streams (long-lived TCP) so the origin tier doesn't sit on 50K+ open sockets, enforces a per-IP global token-bucket BEFORE the gateway. POST requests pass through to the gateway; GET /v1/runs/:id/events terminates here and proxies upstream via Cloudflare's persistent connection pool.

Why it exists. Considered terminating SSE at the gateway; rejected because at peak (~50K concurrent runs × 1 SSE stream each = 50K open connections) Envoy's connection pool exhausts and one bad deploy starves the entire fleet. Edge SSE termination buys us connection multiplexing and, more importantly, a WAF / abuse layer that doesn't share fate with the agent control plane. Cloudflare Workers + the global anycast reach is what makes this affordable; building it in-house is months of engineering for no product gain.

When it fails. Edge degrades → SSE waves disconnect; runs continue server-side (the durable-workflow property holds) but UI shows frozen. Detection: cf_status_5xx_rate, sse_disconnect_rate, gateway_active_streams. Mitigation: client SDK falls back to the long-poll mode of GET /v1/runs/:id/events?mode=poll&since=event_id (event-delta semantics) until SSE recovers. The OpenAI Assistants API stream-interrupt class of bug lands here when the CDN tier is the culprit.

API GatewayEnvoy + Cloudflare WAF + per-tenant token bucket

L7 ingress for every public agent-platform call: POST /v1/runs (start), POST /v1/runs/:id/messages (mid-run user message), POST /v1/runs/:id/cancel, GET /v1/runs/:id, GET /v1/runs/:id/events (SSE — passes through to stream-svc). Validates the JWT against auth, enforces per-tenant token-bucket rate limits BEFORE forwarding so a runaway client can't poison the orchestrator, copies the Idempotency-Key header verbatim onto run-svc, and stamps a request-correlation header (W3C Trace Context) so a single agent run is traceable end-to-end through Temporal activities and tool calls.

Why it exists. Considered baking auth + rate limit inline in run-svc; rejected because (1) a 5xx in run-svc would skip the rate limiter and let the storm reach Temporal — and Temporal's frontend is a finite resource (4 096 history shards is the published ceiling per cluster), (2) per-tenant rate limits are tenancy data we'd otherwise ship to every run-svc replica, and (3) gateway 401/403 responses are constant-time, which run-svc isn't. Centralizing authn behind the gateway means run-svc trusts the signed envelope without a round trip. Stripe's 2019-07-10 retro is the 'rate limit in the wrong layer' anti-lesson.

When it fails. Gateway pool exhaustion masks as 'product is slow' but is upstream load. Detection: gateway_active_streams (50K SSE pass-through is the line), downstream_5xx_rate, per-tenant 429 rate. Mitigation: drain mode + per-zone failover; circuit-break a misbehaving tenant rather than 503'ing everyone. The 'one tenant in a tool loop' pattern (the $47K LangChain incident) presents here first — a tenant's run rate spikes past their bucket, the bucket holds, the cost-worker downstream eventually flips the kill-switch.

Auth / IdentityOAuth 2.1 PKCE + JWKS rotation + Postgres key store

Validates the Authorization header on every public call: hashed key lookup against a sharded Postgres of API keys for service-account auth, OAuth 2.1 + device binding for human users (CLI / IDE installers), JWKS rotation for partner integrations. Returns a signed (merchant_id, tenant_id, scopes, daily_budget_remaining_usd, concurrent_run_quota) envelope back to the gateway, valid 60 s. The same envelope is consumed by run-svc, llm-gw, and mcp-router so each can make scope decisions without an extra hop.

Why it exists. Considered baking authn into run-svc; rejected because (1) a CVE in the run-svc image should not break authn for everyone, (2) restricted scopes are a separate product surface (read-only viewer keys, billing-only keys), (3) the lookup hits a 100M-row keys table that doesn't belong on the agent hot path. Centralizing authn behind the gateway with a 60 s scope cache means the gateway rate-limits before any expensive work, and run-svc trusts the envelope.

When it fails. Auth down → all writes 503 (fail-closed); reads with cached scopes survive 60 s then 401. Detection: auth_5xx_rate, scope_cache_miss_rate. Cloudflare R2 March 21 2025 (credential rotation pushed to dev not prod, then old creds deleted; 100% writes / ~35% reads failed for >1 h) is the cautionary tale for cert/JWKS rotation playbooks.

Service · Run Orchestrator (Write Plane)Go + connect-go to Temporal; Brandur-style idempotency state machine

Hosts every mutating run-lifecycle endpoint: POST /v1/runs (create), POST /v1/runs/:id/messages (signal a mid-run user message), POST /v1/runs/:id/cancel (signal cancel), POST /v1/runs/:id/approve (signal HITL approval). For POST /v1/runs: hashes (tenant_id, request body) into a request_fingerprint, runs the Brandur-style idempotency state machine (claim → started → workflow_signaled → finished), commits a SERIALIZABLE Postgres transaction that writes the run_meta row + the idempotency row in one shot, then issues a Temporal StartWorkflowExecution gRPC call. Returns 201 with run_id and the SSE URL in <500 ms; the agent loop runs asynchronously inside Temporal.

Why it exists. Considered a single combined read+write service. Splitting Run Orchestrator (write) from Stream Reader (read) lets each scale on its own load shape (write QPS ≈ 0.05× read SSE concurrency), isolates blast radius (an SSE storm can't crater run starts), and makes per-endpoint rate limits cleaner. The idempotency state machine MUST live here rather than as a sidecar — Brandur's writeup proves correctness only when state transitions interleave with the business write in the same DB transaction. Considered putting StartWorkflowExecution inside the same SERIALIZABLE txn (2PC across Postgres + Temporal); rejected because Temporal rejects writes during partial-failure and a 2PC commit hold-time would defeat the <500 ms SLO. The outbox pattern (commit run-meta + outbox row, then Temporal-relay drains) is the load-bearing alternative — cleaner failure semantics, +20 ms latency.

When it fails. Idempotency store down → fail-CLOSED 503 (cannot accept a run start without a claim row). Temporal frontend down → run-meta commits, outbox-relay retries SignalWorkflowExecution with exponential backoff; client sees 202 Accepted with a status URL once run_meta exists. Detection: claim_p99 < 50 ms, idem_lock_wait_ms p99 < 100 ms, outbox_relay_lag_seconds < 10. The fail-closed posture is the same Stripe took in 2019-07-10: better to 503 than to silently lose a run start.

Service · Stream Reader (Read Plane)Go + Kafka consumer + S3 transcript reader

Hosts GET /v1/runs/:id/events (SSE) and GET /v1/runs/:id (terminal status snapshot). For SSE: subscribes to the run's Kafka partition (key = run_id) and pushes token / tool-call / status events to the client as Server-Sent Events with monotonic event_id. Honors Last-Event-ID header on reconnect — replays from the partition offset matching that event_id, falling back to S3 transcript archive when the requested offset is older than Kafka retention (24 h hot tier). For status snapshot: cache-aside through Redis (tenant-scoped); falls through to run-meta and (if the run is finished) S3 transcript.

Why it exists. Considered serving SSE from run-svc; rejected because SSE concurrency is ~1 000× run-start concurrency (each run streams for its full duration). A stuck SSE connection pool would back-pressure new runs. Considered making the cache the source of truth for transcripts; rejected because the audit trail must persist for 90 days regardless of cache state. Stream-svc holds NO state — it's pure cache + Kafka consumer + S3 reader. Resumable streams (Last-Event-ID) are non-negotiable: the OpenAI Assistants API's stream-interrupt bug pattern is what happens when the read plane couples run lifecycle to socket lifecycle.

When it fails. Cache cold → status snapshot reads hit run-meta replica → replica saturates → switch to degraded-mode that returns last-known status with a 'may be stale' header. SSE backbone (Kafka) down → existing SSE streams die; clients reconnect with Last-Event-ID; S3 fallback reads keep working but at higher latency. Detection: sse_disconnect_rate, kafka_consumer_lag_ms, sse_active_streams, last_event_id_mismatch_rate.

Service · LLM GatewayRust + Tokio; LiteLLM-protocol-compatible upstream; KV-cache-aware sticky router

Single egress point for every LLM call. Three load-bearing responsibilities: (1) Routing — choose provider (Anthropic / OpenAI / Bedrock / Vertex) per call based on (tenant_id, model_id, region, current error-rate per provider, KV-cache locality); failover when a provider's observed error rate exceeds 30% (NOT when its status page is red — Anthropic Aug 2025 misroute lesson). (2) Caching — sticky-pin requests with the same prompt prefix to the same upstream pod so vLLM/RadixAttention prefix caching hits; for Anthropic, set the anthropic-beta: extended-cache-ttl-2025-04-11 header and cache_control: { ttl: "1h" } because Anthropic silently dropped the default to 5 min in April 2026. (3) Quota enforcement — pre-call hard cap per (tenant, day) and per (run, total): rejects with 402 Payment Required if the kill-switch flag is set on the tenant by cost-worker; rejects with 429 if the per-run token cap is exceeded. The kill-switch is the load-bearing piece: an *alert* is not enforcement.

Why it exists. Considered a thin OpenAI-protocol-compatible passthrough (LiteLLM, Portkey). Rejected because routing has to be cache-aware — Manus's published lesson is that KV-cache hit rate is the single most important production metric, and an OpenAI-protocol passthrough cannot do prefix-locality routing because it has no awareness of the upstream's cache topology. Considered making each service call its own provider directly. Rejected because (1) cost arithmetic would be impossible without a single egress meter, (2) cross-service circuit-break decisions need a shared state, (3) a single PCI-style 'all LLM tokens flow through here' boundary is what makes per-tenant quota auditable.

When it fails. All providers degraded → circuit-break per (provider, region, model); fall back to next-best provider if cross-provider tool-call schema can be normalized (kept normalized in the orchestrator, not provider-specific). Cache-aware routing tier saturates → degrade to round-robin; cache hit rate drops 85% → 60%; cost rises ~3×; pages oncall on cost_per_run_dollars SLO breach (NOT on provider_5xx, which would miss the gray failure). Anthropic Aug 2025 misroute (Sonnet 4 → 1M-context servers) is the canonical 'gateway routes by capacity tier, not by SLI' anti-pattern.

Service · MCP Tool RouterTypeScript + MCP SDK; OAuth 2.1 PKCE; per-tool circuit breaker

Single egress point for every tool call (MCP servers + first-party tools). Validates the (run_id → allowed_tool_set → required_oauth_scope) capability triple before any call lands at a third-party server, runs a per-tool circuit breaker that opens after 50% errors over 30 s, terminates the OAuth 2.1 flow with PKCE for user-bound tools, caches tools/list per server (with cache_tools_list=true because regenerating the tool catalog every call is the published MCP scaling foot-gun). Returns the tool result to the caller, redacted by the per-tenant output classifier (SecretRedactor + PIIRedactor) because tool output is the prompt-injection attack surface.

Why it exists. Considered letting the orchestrator activity workers call MCP servers directly. Rejected because (1) the prompt-injection 'lethal trifecta' (private data + untrusted content + external comms — Simon Willison) is broken exactly here: a single egress point can scrub tool output before it reaches the next LLM call, (2) per-tool kill-switches must work fleet-wide, (3) the EscapeRoute Anthropic filesystem MCP CVE-2025-53109/53110 (path-traversal) is the kind of bug that needs a defense-in-depth boundary the agent runtime CANNOT bypass. Considered making capability scoping a sandbox-pool concern (egress proxy in the microVM); rejected because some tool calls happen from orchestrator activities outside any sandbox.

When it fails. Per-tool circuit open → that tool returns 'unavailable' to the agent; the model sees a clean error and pivots in the prompt. Classifier stalls → fail-CLOSED on tool output (better to drop a tool call than to leak). Detection: per-tool circuit_open_rate, classifier_p99, oauth_token_refresh_failure_rate. The Anthropic April 23 2026 Claude.ai MCP-applications outage (00:41-02:09 UTC, v2.1.120 force-rolled-back) is the precedent — a bad MCP layer rollout starves agents fleet-wide, but if circuit breakers fire per-tool, only impacted tool flows degrade.

Worker · Sandbox PoolFirecracker microVMs + KVM + virtio-net + memory-snapshot fork

Hosts the per-task Firecracker microVMs that run agent-generated code, headless browsers (Chromium + Playwright), and untrusted tool-output processing. Each run gets a dedicated microVM forked from a warm snapshot (~150 ms with Firecracker per E2B's published numbers; ~10× faster than fresh boot). Inside the VM: 2 vCPU / 1-2 GB RAM, ext4 root, a unix-domain-socket egress proxy that enforces the per-tenant capability token, /workspace mounted from a per-run scratch volume. Acts as a Temporal activity worker for bash, code_exec, browser_action, read_file, write_file activities. On task end (or eviction), the microVM is snapshotted to S3 and torn down.

Why it exists. Containers (runc) share a kernel, and Cognition's published 'Building Cloud Agents' post documents the exact failure: 'a single compromised session can reach every other container's filesystem.' Firecracker microVMs (KVM hardware virtualization) are the production isolation primitive every coding-agent vendor (E2B, Cognition, Vercel, Devin) converged on. Considered gVisor (user-space kernel, weaker but cheaper cold-start) — chose Firecracker because runc Leaky Vessels (CVE-2024-21626) and the Nov 2025 trio (CVE-2025-31133/52565/52881) prove that container-class isolation is too thin for arbitrary agent-generated code. Considered always-on warm pool only — the published cost crossover with fork-from-snapshot is around 10 sandboxes/sec arrival rate (E2B blog); we run hybrid (warm pool absorbs steady-state, fork-from-snapshot absorbs bursts).

When it fails. Warm pool exhaustion → new runs queue; sandbox-acquire p99 climbs from sub-second to tens of seconds. Detection: sandbox_queue_depth, warm_pool_size, fork_p99. Mitigation: spillover to cold-start tier + emergency host autoscale (autoscale lag 2-3 min). The E2B Mar 9 2026 'Sandbox management degraded' incident is the precedent. Sandbox escape via runc Leaky Vessels-class CVE → quarantine the affected host, reroute to the rest of the pool (1/256 blast radius), forensics on the host. Anthropic's filesystem MCP EscapeRoute (CVE-2025-53109/53110) is the closest agent-platform-specific precedent.

Coordinator · Temporal ClusterTemporal Server 1.27 (frontend + history + matching + worker, multi-cluster replication)

Owns every agent run as a durable workflow keyed by run_id. The workflow code is deterministic Go that drives the agent loop: plan → call LLM (activity) → call tool (activity) → update memory (activity) → repeat. LLM and tool calls are non-deterministic Activities with at-least-once semantics, idempotency keys derived from activity_id, and exponential-backoff retries (5 attempts default, configurable per activity). Workflow history is event-sourced into Cassandra (the workflow-history node) and replayed on any worker restart. Signals (cancel, user-message, HITL approve) are first-class — they're how the read plane and the user influence the run without breaking determinism. Pointer State pattern: large activity payloads (full LLM responses, tool results) are written to S3 and only an S3 pointer lands in the history (99.8% payload reduction per AzGuards' published guidance — keeps history shards small).

Why it exists. The single insight that unlocks this whole design: an agent run is a durable workflow, not a request. LangGraph 1.0 ships checkpointing per super-step, but Diagrid's published critique 'Checkpoints Are Not Durable Execution' shows it lacks at-least-once activity semantics, exponential backoff, and signal-based interrupts that Temporal provides. Cognition's Devin runs on Temporal; Replit Agent 3 runs on Temporal (case study published); OpenAI Codex web agent runs on Temporal. Considered DBOS (Postgres-resident workflow state) and Restate (HTTP sidecar); both are viable, but Temporal's Cassandra-backed history shards and proven 4 096-shard cluster ceiling are what we need at 50 K concurrent runs. The non-trivial trade-off accepted: workflow code MUST be deterministic — every replay produces the same control flow given the same history. This forces all I/O into activities, all randomness into Workflow.NewRandom, all time into Workflow.Now — and any new agent prompt version must gate via Workflow.GetVersion or it explodes on replay (the non_deterministic_error pager).

When it fails. History shard quorum loss → workflows on those shards stall (read+write block); other shards continue. Frontend down → run-svc StartWorkflowExecution returns 503; outbox-relay retries; user sees brief delay. Worker pool down → activities don't dispatch; in-flight workflows block on activity ScheduleToClose. The most insidious failure: non_deterministic_error after a deploy where workflow code changed without a Workflow.GetVersion gate; in-flight workflows fail-closed on replay. Detection: temporal_workflow_task_failed{cause='non_deterministic_error'} > 0 is a P1 page; shard_lock_p99, history_event_persist_p99.

NoSQL DB · Workflow HistoryCassandra 5.0 (Temporal default backing store)

Cassandra cluster that backs Temporal's workflow history. Two main tables: executions (per-run pointer to current history slice) and history_node (the append-only event log per workflow_id). Every workflow event — workflow started, activity scheduled, activity completed, signal received, timer fired — is appended here. The event log IS the source of truth: any worker can replay a workflow from event 0 and arrive at the same state. Partition key = workflow_id (run_id); clustering key = event_id (monotonic per workflow).

Why it exists. Considered Postgres for the history (Temporal supports it for small clusters); rejected because at 50 K concurrent runs × ~500 events / run / hour = 25 M events/hour, Postgres' WAL fan-out becomes a bottleneck around 5-10 K writes/sec. Cassandra's leaderless quorum scales horizontally on writes and partitions cleanly by workflow_id. Considered FoundationDB (some Temporal forks use it); rejected because Cassandra is the documented production-default with the largest community of operators. Considered S3 for history (event-sourced into object store); rejected because per-event read latency is fatal at ~500 events/run replay.

When it fails. Quorum loss (2 of 3 AZs partitioned for one shard's replicas) → workflows on that shard halt; W cannot form, R cannot form. Detection: temporal.persistence.error_rate, cassandra.write_timeout, cassandra.read_timeout. Monzo 2019-07-29 (auto_bootstrap=false on six new nodes returned empty quorum reads) is the canonical retro — the lesson is that scale-up tests must include multi-node bootstraps, not just single-node. Mitigation: hinted handoff queue depth alert at 1 GB; never relax to ONE on writes.

SQL DB · Run Metadata + Idempotency + Vector MemoryPostgres 16 + Citus + pgvector + pgvectorscale

The relational source of truth for: (1) runs table (run_id, tenant_id, workflow_id, status, tags, created_at, terminal_at, billing_cents) — the index every list / search query hits; (2) idempotency_keys table (Brandur-style, 24 h hot + 48 h soft-retain TTL); (3) tenant_meter (running cost-per-day, run-count-per-hour, kill_switch_engaged_at — the row cost-worker writes to); (4) episodic_memory table with pgvector embeddings for cross-run retrieval (tenant-scoped, recall@10 target 0.92). Sharded by tenant_id on Citus (32 shards) so per-tenant queries are single-shard. The SAME engine hosts idempotency and runs because the run_meta INSERT and the idem CLAIM commit in one SERIALIZABLE transaction at run-svc.

Why it exists. Considered DynamoDB for runs+idempotency (single-key conditional writes are perfect for CLAIM); rejected because the runs table has rich query patterns (list by tenant, filter by status, paginate by created_at) that DynamoDB models awkwardly. Considered separating idempotency into Redis; rejected because the idempotency contract requires state transitions atomic with the business write. Considered a dedicated vector DB (Qdrant, Turbopuffer) for episodic memory; for the default scale tier (≤10 M vectors per tenant) pgvector + pgvectorscale gives 471 QPS @ 99% recall on 50 M vectors (TigerData benchmark) and keeps memory co-located with run metadata — the scale-out path to Turbopuffer is documented but not the default.

When it fails. Single shard primary down → 1/32 of tenants see write failures for ~30 s failover. Cross-shard read fan-out for admin / billing queries hits multiple shards in parallel; one slow shard slows the query (head-of-line). Detection: per-shard write_p99, replica_lag_seconds, oldest_unreplicated_lsn, idempotency_lock_wait_p99. Brandur's writeup is explicit: idempotency-store fail-closed is mandatory. Adjacent risk: episodic_memory is a cross-run prompt-injection vector — a malicious tool result that gets embedded and recalled on a future run by the same tenant pollutes downstream context; the output classifier runs on tool returns but a second-stage classifier on memory writes is the load-bearing mitigation (NOT shipped in v1; tracked as accepted residual risk).

Stream · Event BusKafka 3.7 (KRaft mode), 256 partitions, RF=3, acks=all

The async backbone connecting llm-gw, mcp-router, sandbox-pool, run-svc as producers to cost-worker, stream-svc, and the OTel trace sink as consumers. Three logical topics: (1) agent.events.v1 — every coalesced token chunk, llm.call, tool.call, sandbox.lifecycle, run.status; partitioned by run_id (preserves SSE order for stream-svc). (2) cost.meter.v1 — per-call cost samples; partitioned by tenant_id (preserves cost-worker single-writer per tenant). (3) audit.v1 — security-audit events (capability-token-issued, kill-switch-fired, cert-rotation-completed); partitioned by tenant_id, RF=5, immutable. Producers use the idempotent producer with acks=all + min.insync.replicas=2 = exactly-once into Kafka.

Why it exists. Considered direct service-to-service calls (llm-gw → cost-worker via gRPC). Rejected because (1) cost enforcement needs a durable backplane — losing a cost event means the meter under-counts and the kill-switch fails late; (2) SSE fanout to many concurrent stream-svc replicas needs broadcast semantics; (3) the audit topic is the load-bearing piece for compliance — every capability-token issuance and every kill-switch fire is a regulator-readable event. Considered Pulsar (better tenancy isolation); chose Kafka because the operator pool is larger and the published Cassandra+Kafka+Temporal stack is a known-good shape. Considered NATS JetStream; rejected because the durability guarantees aren't acks=all+ISR equivalents at this scale.

When it fails. ISR shrinks below min.insync.replicas → producers block; llm-gw and sandbox-pool back-pressure; SSE token streams pause. Two-broker loss in same AZ stops producers for that AZ. Detection: under_replicated_partitions, isr_shrink_rate, producer_request_latency_p99. Mitigation: 3-AZ broker spread is non-negotiable; per-AZ producer affinity so a single-AZ broker outage is degraded-not-broken.

Worker · Cost Aggregator + Kill-SwitchGo + Kafka consumer + Postgres SELECT FOR UPDATE

Consumes cost.meter.v1 from Kafka, aggregates per-tenant cost in 1-second buckets, writes the running cost into tenant_meter in run-meta. When (cumulative cost / day > tenant_budget) OR (cost / minute > runaway_threshold) OR (no-progress detector — same tool args repeated >5 times in 60 s), engages the kill-switch: writes kill_switch_engaged_at = now(), publishes a control signal to llm-gw via gRPC (cancels in-flight + rejects new calls for that tenant), and pages on-call P1.

Why it exists. Considered enforcing the cap inside llm-gw directly — cumulative state per tenant lives in Redis, every call decrements. Rejected because (1) Redis isn't durable (a flush would lose the meter), (2) cross-replica consistency on the meter is required (32 llm-gw replicas all writing to the meter row would cause hot-row contention), (3) the no-progress detector is a windowed-stream computation that doesn't fit at the gateway. Centralizing into a single-writer-per-tenant cost-worker (Kafka partition by tenant_id ensures this) gives us atomic-in-the-stream meter updates. The $47K LangChain incident retro is the lesson: an open-loop billing alert is not enforcement — the kill-switch must fire at minute granularity, not at the daily billing job.

When it fails. Cost-worker consumer lag > 60 s → meter updates stale → kill-switch fires late → tenant overspends past their cap. Detection: kafka_consumer_lag_seconds (P1 at >30 s), kill_switch_decision_latency_p99, runaway_detected_count. Mitigation: scale-out via cooperative rebalance (no stop-the-world); fallback hard-cap at gateway (1.5× tenant budget) so even a fully-failed cost-worker bounds the loss. Replit users reported $30/hr → $360/day before their 2025 controls landed; this node prevents that.

Object Store · Transcripts + SnapshotsS3 + Object Lock + Glacier tier

Cold-tier and audit-grade store for: (1) per-run full transcripts (every LLM call request+response, every tool call, every sandbox stdout/stderr) keyed runs/{tenant_id}/{run_id}/transcript.jsonl; (2) sandbox memory snapshots from Firecracker for resume-on-crash, keyed sandbox/{run_id}/snap-{seq}.bin; (3) Pointer State payload offloads from Temporal (large LLM responses, tool results) keyed temporal/{workflow_id}/payload-{event_id}.bin; (4) audit.v1 long-term sink with Object Lock (compliance-grade WORM for 7 years). Multi-region replication to a DR bucket; Glacier tier for runs > 90 days old.

Why it exists. Considered keeping transcripts in run-meta (Postgres); rejected because per-run transcript can hit 100 MB on long Cursor / Devin sessions and Postgres TOAST + WAL volume would be catastrophic at scale (AzGuards documented the 99.8% payload reduction from offloading). Considered DynamoDB for transcripts; rejected on cost (S3 is ~25× cheaper for cold reads). The Pointer State pattern (write the heavy payload to S3, only the S3 URL into Temporal history) is what keeps Temporal history shards small and replay fast.

When it fails. S3 regional event → reads fail; resume-on-crash slows (sandbox snapshot unreadable) → microVM cold-starts from base instead of from snapshot. Detection: s3_5xx_rate, snapshot_load_p99, transcript_write_failure. Mitigation: cross-region replication with DR-region active read fallback; Pinecone March 2023 (515 indexes deleted by buggy SQL cleanup) is the cautionary tale for any data-store with bulk-delete capability — Object Lock COMPLIANCE prevents the equivalent on audit data.

External · LLM ProvidersAnthropic Claude API, OpenAI, AWS Bedrock, Google Vertex

Third-party LLM inference. Anthropic Claude (Opus 4.7 / Sonnet 4.6 / Haiku 4.5) for the primary route, OpenAI GPT-5 / o3 for fallback, Bedrock for VPC / data-residency-bound tenants, Vertex for GCP-native customers. Each provider's tool-call format is provider-specific (Anthropic tool_use blocks vs OpenAI tool_calls arrays); the orchestrator stores normalized form so failover is possible mid-run. Streaming responses come back via SSE on the provider side; llm-gw relays into the event-bus.

Why it exists. Considered self-hosting (vLLM + open-weight Llama / Mixtral / Qwen). Rejected for the default canonical because frontier-model quality (Claude Opus 4.7 / GPT-5) is the load-bearing piece for agent reliability, and no open weight matches it as of 2026. The KV-cache-aware llm-d / Nvidia Dynamo path is documented for the vendor-lock-mitigation tier (large enterprises) but not the default. Considered single-provider; rejected because of the OpenAI 2023-11-08 (94 min outage), Anthropic April 2026 (multi-hour Claude.ai/API/Code outage), and Anthropic Aug 2025 misroute (Sonnet 4 → 1M-context servers, 16% degraded) precedents — a multi-hour single-provider outage takes the entire platform down without multi-provider routing.

When it fails. Provider hard down → multi-provider failover; tenants whose tool-call format is provider-locked may fail mid-run (mitigation: keep normalized form in orchestrator, not provider-specific). Provider gray failure (5xx on a subset of model+region pairs while status page is green) → circuit-break per (provider, region, model) on observed error rate. Anthropic Aug/Sep 2025 postmortem documents three concurrent provider issues that mainstream multi-provider failover masked partially. OpenAI 2024-06-04 5+ hr ChatGPT outage is the precedent for hard-down.

External · MCP ServersAnthropic-spec MCP (OAuth 2.1 PKCE for remote)

Third-party MCP (Model Context Protocol) servers exposing capabilities to the agent: filesystem MCP, GitHub MCP, Linear MCP, Slack MCP, browser-use MCP, and per-tenant custom MCPs. Each server advertises a tool catalog via tools/list; the agent calls a specific tool via tools/call. Servers act as OAuth Resource Servers per the June 2025 MCP spec, advertising their auth endpoints via .well-known. Mediated entirely through mcp-router so the platform sees every tool call and can scope/filter/audit.

Why it exists. Considered first-party tool implementations only. Rejected because the MCP ecosystem hit 97 M monthly SDK downloads / 81 K GitHub stars by March 2026; building every integration in-house is non-leverage. The trade-off accepted: third-party MCP servers are an attack surface (Anthropic filesystem MCP CVE-2025-53109/53110 EscapeRoute, Anthropic Git MCP injection flaws fixed Jan 2026) — every MCP must be sandboxed by capability scoping and per-tool circuit breakers at mcp-router, NEVER directly exposed to the model.

When it fails. MCP server hard-down → per-tool circuit-breaker opens; agent sees 'tool unavailable' and pivots in the prompt. Compromised MCP server (supply-chain / CVE) → capability scoping limits blast radius; output classifier blocks exfil attempts. The Anthropic April 23 2026 'MCP applications on Claude.ai unavailable' (00:41-02:09 UTC, v2.1.120 force-rolled-back) is the precedent for fleet-wide MCP layer failure. EscapeRoute (CVE-2025-53109/53110) is the precedent for compromised-server-as-attack-vector.