Web Crawler — a worked solution

Typical clarifications to surface:

• Scope: Whole-web (Common Crawl shape) or vertical (e.g. e-commerce only)? Bounds the seed list and the abuse-tolerance.
• Freshness vs. coverage: Is the goal "discover every URL once" (archival) or "keep the top 1 M sites fresh" (search-engine-shape)? Re-crawl scheduling looks completely different.
• JS rendering: Index SPAs? If yes, the renderer fleet dominates compute cost.
• AI-training corpus or live-search? Determines retention, dedup tolerance, and whether you publish the WARC archive.
• Operator constraints: Take-down SLA? Verifiable identity (reverse-DNS + published IP allowlist) required to avoid Cloudflare AI Audit blocking?
• Multi-region: Single-region keeps the frontier simple; multi-region requires partitioned-by-host frontiers + cross-region gossip for discovery.

Assumptions to state:

• 10 B URLs/month = ~3.86 K sustained fetch QPS, ~7.7 K peak (2× burst).
• 100 M hosts in scope; politeness floor 1 RPS/host, average sustained 0.05 RPS/host (long-tail dominated).
• 20 % of pages need JS rendering.
• Single AWS region (us-east-1); colocate everything with S3.
• 5-year WARC retention.
• Public, identified crawler — verifiable UA + reverse-DNS, IP allowlist published at /.well-known/crawler-ip-ranges.

• Submit seed URLs / sitemap pulls (operator API, low QPS).
• Discover new URLs from parsed pages and enqueue if not seen.
• Fetch URLs respecting per-host / per-IP / per-AS politeness windows + robots.txt rules.
• Render JS-required pages via headless Chromium pool.
• Archive fetched bytes to WARC in S3.
• Index parsed text/metadata into the search index.
• Take down a host or URL pattern within < 1 hour of operator command.
• Re-crawl by adaptive schedule (change-rate driven, not size driven).
• Publish identity (UA, IP allowlist, public robots.txt for our agent).

• Crawl freshness (top 1 M sites) p99: < 24 h since last successful fetch.
• Robots.txt compliance: 99.99 % — every miss is a paged on-call incident with legal escalation.
• Politeness violation rate: < 0.01 % of fetches exceed configured per-host RPS.
• Fetcher availability: 99.9 % — drops are acceptable (the freshness SLO is what matters), but cascading freezes are not.
• Frontier durability: 99.99 % within-region (~52 min annual budget) — single-region multi-AZ Cassandra at QUORUM with 5-min Bloom snapshots and 15-min Cassandra incrementals. Five-nines is reserved for cross-region crawlers that the next inflection-point upgrade will deliver; we explicitly do not claim 99.999 % on a single-region design with a 12 h restore RTO.
• Indexing pipeline lag p99: < 5 min fresh, < 1 h backfill.
• Take-down SLA: < 1 h global propagation, audit-logged.
• Cost ceiling: Render fleet is the dominant variable cost; cap it at 30 % of total infra spend or scale the render fraction down.

Mid-tier walk-through. All numbers derived; alter the inputs in the simulator to re-derive.

Fetch QPS sustained = 10B / (30 × 86400)        = 3.86K  qps
Fetch QPS peak      = 3.86K × 2                  = 7.72K  qps

Concurrent hosts in flight. Politeness caps each host at 1 RPS, but the average polite host sustains ~0.05 RPS (long-tail wordpress sites). So:

Concurrent hosts = 7.72K / 0.05 = 154K hosts in flight at peak

That number is the design constraint. Heritrix tops out around 250 ToeThreads per process, but on a 2025 c6a.4xlarge (32 vCPU/64 GB) with HTTP/2 multiplexing we run 256 ToeThreads/pod conservatively. We size for 30 % headroom over the median demand because tail events (slow-host clusters) compound: 800 fetcher pods × 256 ToeThreads = 204 K slot capacity vs 154 K demand = 32 % cushion. 10 % of slots reserved for a "long-tail" subpool that handles hosts with historical p99 > 10 s, so slow hosts don't saturate the main fleet. The crawler is bound by host count, not raw fetch capacity.

Bloom filter for the seen-set. 100 B known URLs at 0.1 % FPR:

bits/URL = -ln(0.001) / (ln 2)^2 ≈ 14.4 bits ≈ 1.8 B
total    = 100B × 1.8 B          = 180 GB sharded
shards   = 32 (per-shard 5.6 GB, fits in RAM in front of the Cassandra journal)

robots.txt fetches. 100 M hosts / 24 h = 1.16 K req/s — 15 % of page-fetch QPS. Non-trivial; needs its own cache and dedicated robots-fetch headroom.

WARC ingest & storage.

ingest_MB/s = 3.86K × 25 KB        = 96 MB/s
daily       = 8.3 TB
monthly     = 250 TB
5y archive  = ~15 PB compressed

Render fleet. 20 % render-required × 7.72 K peak = 1.54 K renders/sec. At 5 s avg duration: 7.7 K concurrent renders. Chromium mean RSS is ~600 MB but p99 is 1.5–2 GB; the 1 GB per-tab cap is a kill threshold, not the mean. Sized for p99 burst: 400 c6a.4xlarge × 24 renders = 9 600 concurrent capacity vs 7 700 demand = 25 % headroom. ≈ $179 K/month (dominant compute cost). Bounded queue depth 5 K with overflow-policy = drop-to-raw-html so a render saturation degrades the index gracefully instead of cascading.

Kafka pages-stream. 7.72 K msg/s peak × 32 KB envelope (gzipped page + headers + URL + envelope) = 245 MB/s through the broker. 512 partitions sized for two-level keying (host + url-bucket) so a hot host like Wikipedia doesn't pin a single partition — the power-law p99/p50 of 5–10 × in flat hash-by-host needs the extra partitions plus the bucket dimension to stay inside per-partition 60 MB/s ceilings. RF=3 + min.insync=2 + acks=all gives RPO = 0 across an AZ loss.

Why 16 frontier shards / 32 seen shards. Frontier writes are dominated by parser discovery (~5–10× the fetch QPS in URLs/s but tiny payloads); 16 shards × 1 K writes/sec/shard fits comfortably in Cassandra at quorum. Seen-set is much larger (180 GB working set) and reads are RYW for parser self-checks → 32 shards isolates the hot edge of the keyspace.

Why these timeouts. Fetcher → target-web at 30 s = the hard ceiling (industry-standard, matches Googlebot). Fetcher → renderer 35 s > renderer → target-web 25 s so the cascading-retry rule (callee_total_budget < caller_per_attempt_timeout) holds and the renderer is allowed to fail-fast inside the fetcher's window. Fetcher → pages-stream at 5 s with acks=all+fixed-retries = the producer DOES await ack; on Kafka unavailability the fetcher writes to a local disk-backed outbox (60 s capacity) so the slot is freed and the page is not dropped. Scheduler → fetcher at 100 ms = the dispatch lease ack budget; the fetch itself is async after the ack. Cascade chain: scheduler 100 ms > fetcher-ack ~5 ms ✓; fetcher 30 s > publish 5 s × 3 retries = 15 s ✓ but the local outbox absorbs publish failures so the slot SLA holds at fetch+15 s = 45 s p99 worst-case; scheduler dispatch loop tolerates this because slots are not slot-pinned-per-URL but slot-pinned-per-fetch.

POST /api/v1/seeds
Authorization: Bearer <op-token>
{ "urls": ["https://example.com/", "https://example.org/sitemap.xml"], "priority": "normal" }
202 Accepted        # gw → scheduler (e3); scheduler normalizes + enqueues into the frontier (e15b)

POST /api/v1/takedown
{ "scope": "host", "value": "example.com", "reason": "operator-request", "ticketId": "TKT-1234" }
202 Accepted        # propagates to scheduler + robots cache within 1h

GET /api/v1/freshness?host=example.com
200 { "host": "example.com", "lastFetchedAt": "...", "knownUrls": 1834221, "fetchedThisMonth": 18234 }

# Public crawler-identity surface (unauthenticated, CDN-cached):
GET /.well-known/crawler-ip-ranges
200 { "ranges": ["203.0.113.0/24", ...], "userAgents": ["ArqlyBot/1.0"], "verifyDoc": "https://arqly.example/bots" }

Frontier (Cassandra, 16-shard, RF=3):

Per-host quota (Cassandra counter table — a counter column can't share a table with non-counter columns, so the per-host cap lives beside the frontier, not inside it):

Seen-set (Cassandra journal beneath an in-memory Bloom):

robots.txt cache (Redis hash):

Database choice — recommended. Cassandra (or ScyllaDB) for both frontier and seen-set. Why: leaderless replication survives an AZ loss without a write outage; quorum tunability lets the seen-set serve RYW reads from the parser without paying linearizable cost on the frontier dequeue. Discord's trillion-message ScyllaDB migration shows the pattern at write-heavy scale. Postgres alone falls over at 100 B-row dedup tables; DynamoDB works but locks you to AWS and the per-row pricing on a 100 B-row table is brutal. Redis is the right cache for robots.txt (24 h TTL, 200 GB working set) but not the system of record. S3 for WARC is uncontroversial — it's what Common Crawl and Internet Archive use, with 11×9 s durability built in.

Architecture summary — the three loops:

1. Crawl loop (hot path). Scheduler pops a host whose politeness window has elapsed → leases the URL to a fetcher pod → fetcher checks robots cache → fetches the page (or hands off to renderer for SPAs) → publishes the raw fetch to pages-stream. The fetcher never blocks on a downstream commit — the parser side runs independently.
2. Parse-and-commit loop. Parser consumes pages-stream → extracts links → SimHash for near-dup detection → outbox-batch commit (seen + frontier in one Cassandra batch keyed by host) → durable PUT to WARC → async batch publish to the search index. The outbox is the load-bearing atomicity primitive: seen + frontier must never diverge, or you re-crawl forever (or never at all).
3. Operator/admin loop. Operator submits seeds and takedowns through the dashboard → CDN → Gateway → Scheduler. Takedown propagates to the scheduler's politeness controller AND the robots cache as a synthetic Disallow within < 1 h. Seed injection: the scheduler validates + normalizes the seed and enqueues it into the per-host frontier subqueue (e15b, scheduler → frontier, op: write) — the same normalize+enqueue terminus the parser uses on discovery (e15). A seed routed gw → scheduler (e3) therefore has a real frontier terminus: it becomes a frontier row exactly like a discovered URL. The seed enqueue is a single best-effort write (no page-commit txn); only the parser's discovery write (e15) is part of the outbox group, since only there must seen + frontier commit atomically.

Why the Mercator back-queue. The classic 1999 design — per-host FIFO subqueues with a back-queue selector picking the next ready host — is still the right primitive. It enforces the per-host concurrency invariant as a data-structure property, not a runtime check. Heritrix's BdbFrontier, StormCrawler's ES-status-index, and Googlebot's "Jack" all descend from this. The scheduler shards by hash(host) so politeness state stays local and a 12-pod scheduler scales horizontally.

Why Bloom + Cassandra journal for the seen-set. A naive all-Cassandra seen-set requires 100 B point reads per discovery batch (10× fan-out of the fetch rate) — that's 80 K reads/sec on the parser side, untenable at quorum latency. The Bloom filter cuts 99.9 % of those before they hit the journal, and the Cassandra journal serves the remaining false-positive lookups in single-digit ms at quorum. The 0.1 % FPR is the right knob: a false positive means we don't enqueue a URL we haven't seen, which is mild data loss in a fundamentally best-effort archival pipeline. False negatives (saying we haven't seen something we have) are impossible — that's why Bloom over hash-set is correct here, not the other way around.

Why fail-CLOSED on robots.txt parse error. RFC 9309 is strict; mis-served HTML-as-robots is a real failure mode. Treating a parse failure as "Allow: /" trades 24 h of compliance violation for a few minutes of operator pain. Treating it as "Disallow: /" risks dropping a few thousand crawlable hosts but bounds legal exposure. Production crawlers (Google, Bing, Common Crawl) all fail-closed; we follow.

Why outbox, not 2PC, on the page commit. The parser writes seen + frontier in a single Cassandra batch (single shard, both keyed by host) PLUS an outbox row in the same batch — that is the only atomic step (mode: outbox, participants: dedup + enqueue). The outbox row is then drained by the parser to: (a) WARC PUT (sync, exp-backoff with max 5 retries before sideband to a warc-failed table), and (b) bulk-publish to the index ingester. WARC is not a participant in the page-commit txn — it is downstream of the outbox drain, so a WARC PUT failure cannot un-commit seen+frontier and a successful seen+frontier commit cannot block on WARC. This matches the JSON: only e14 (parser→seen) and e15 (parser→frontier) carry txn.group=page-commit; e13 (parser→warc) and e16 (parser→index) are downstream propagators. 2PC across Cassandra and S3 is impossible in practice (S3 has no participant API); saga is overkill since the compensating action is "the next discovery re-enqueues the URL and we re-fetch," which is not a true compensation but the natural retry semantic. The accepted trade-off: a crash between the Cassandra batch commit and the WARC PUT loses that page's archive (not the index entry, which is rebuilt on next re-crawl). Outbox-lag dashboard catches systemic versions.

Why per-host AND per-IP AND per-AS politeness. A naive per-host limiter explodes against shared hosting (100 K vhosts on one /24). Cloudflare's published per-AS observations show the long tail of crawler abuse: most polite-per-host crawlers are accidentally rude per-AS. Three-tier token bucket in the scheduler is the load-bearing fix. Coordination across the 12 scheduler shards is centralized in a separate Redis cluster (one logical cluster, shared by all scheduler shards) keyed by (tier, key) where tier ∈ {host, /24, AS}. Per-shard local cache absorbs reads with 100 ms TTL; writes (decrement) go to the central Redis with optimistic concurrency. Without this central coordination, scheduler shard 5 and shard 7 would each grant 1 RPS to AS-12345 → 2× the per-AS budget. The Redis cluster is RDB-persisted with 5-min snapshots so a restart doesn't reset every bucket to "full" (which would permit a politeness-regression burst).

Takedown control plane (< 1 h SLA). The takedown propagation is engineered, not asserted. (1) Operator submits to gw with bearer + ticket-system reference. (2) gw writes synchronously to a Kafka compacted topic takedown-bus (RF=3, retain forever) AND issues control RPCs to the scheduler (immediate suppression on dispatch) and to the robots cache (synthetic Disallow: / injection for the targeted host). (3) Fetchers re-pull from the robots cache on every fetch (no fetcher-resident takedown table) so robots-cache-injection alone covers the fleet within one robots-TTL refresh. (4) Audit log writes to takedown-bus with timestamps for every step; the legal team gets a runbook of "submission → propagation → confirmation". Worst-case propagation budget: gw→scheduler RPC (1 s) + gw→robots RPC (1 s) + 600 fetcher pods discovering the takedown on next robots-cache-refresh (worst case = robots TTL = 24 h, but typical = next fetch within seconds). To meet the < 1 h SLA the robots cache injection is the load-bearing step (immediate); the scheduler suppression is a defense-in-depth.

Multi-region posture. Single-region (us-east-1, colocated with S3) for this canonical. Cross-region adds: partitioned-by-host frontier with cross-region gossip for discovery (each region owns a host-shard and discovers URLs locally; a small cross-region delta replicates "newly discovered" deltas every 30 s). For an SRE proposing global active-active, the burden is showing that the cross-region egress + politeness coordination is worth it; for most use-cases, single-region with multi-AZ for HA is the right call.