Distributed Cron — Mass Scheduled Email

Single trigger, 50M recipient idempotency, catch-up.

SavedSaved on this device

The problem

Single trigger, 50M recipient idempotency, catch-up.

A campaign-owner uploads a 50-million-recipient list and schedules a single send for 9:00 AM tomorrow. At 9:00 AM, one scheduler tick fires — and that single event must become 50 million emails leaving SES within 60 minutes, with the strict invariant that no recipient receives the same campaign twice, ever, regardless of scheduler crashes, region failovers, Kafka rebalances, fan-out worker deaths, SES throttles, or operator-initiated catch-up replays.

This is the simplest-sounding distributed-systems problem that turns out to be one of the hardest. Three things define it: (1) at-least-once everything except the very tail — the scheduler, the fan-out controller, the Kafka pipeline, every retry, every catch-up replay is at-least-once because making them exactly-once is intractable; (2) a durable per-recipient idempotency tail that collapses at-least-once into at-most-once-per-recipient at the SES send boundary; (3) a rate-paced catch-up controller that knows the difference between "fire the missed run now" and "the email is too stale to send" — a 9 AM "good morning" email arriving at 1 PM is reputation damage, not delivery.

The architecture comes straight from Google's Borgcron paper (Sundheim, ACM Queue 2015) for the scheduler tier, LinkedIn ATC and Pinterest NEP for the fan-out tier, Stripe / Brandur's idempotency-key contract for the dedupe tier, and DoorDash's Cadence-as-fallback pattern for the catch-up reconciler. Every load-bearing decision below has a published precedent — this is not speculation.

Reading: Sundheim — Reliable Cron across the Planet (ACM Queue 2015, Borgcron paper) · LinkedIn — Air Traffic Controller: Member-First Notifications (2016) · LinkedIn — Hermes mass email retros (eng blog) · Pinterest — NEP Notification System and Relevance (Medium 2017) · Uber — Cherami: Uber's durable distributed task queue · Uber — Announcing Cadence (Temporal predecessor) · DoorDash — Cadence as a Fallback for Event-Driven Processing · Discord — How we store trillions of messages (Cassandra → ScyllaDB) · Stripe — Designing robust idempotency keys (Brandur) · Brandur — Implementing Stripe-like Idempotency Keys in Postgres · AWS Builders' Library — Idempotency at Scale (re:Invent ARC403 2021) · AWS — Handling SES throttling (Maximum sending rate exceeded) · AWS — SES sending quotas & dedicated IP warmup · Mailgun — Bulk email sending with queue management · Slack — Tracing notifications (Go → Kafka → Elasticsearch) · Shopify — High availability background jobs · Vallery Lancey — Kubernetes CronJob Failed For 24 Days (case study against naive cron) · Quartz Scheduler — JDBC JobStore clustering docs (the misfire-threshold story) · Cloudflare — Cron Triggers internals (Nomad-distributed schedulers) · Apache Kafka — KIP-429 cooperative incremental rebalancing · Beyer et al. — SRE Workbook ch. 21–23 (overload, cascading failures, critical state) · Campbell & Majors — Database Reliability Engineering ch. 9 (fencing tokens)

Borgcron-style at-least-once scheduler
per-recipient idempotency at fan-out tail
outbox (durable trigger log + publish)
saga (campaign create + bulk recipient upload)
active-region fencing token
two-tier dedupe (hot Redis + durable Cassandra)
rate-paced catch-up reconciler
per-tenant Kafka bulkheading
SES warm-pool reputation isolation
TCPA opt-out linearizability

01Clarifications

What would you ask before drawing a single box?

Ambiguity you would resolve with the interviewer: scope, scale, who uses it, what counts as done.

About Distributed Cron — Mass Scheduled Email

Single trigger, 50M recipient idempotency, catch-up.

Difficulty: advanced
Time: about 75 minutes
Stages: 10
Topic: Queues, Pub/Sub & Event Streaming

How this problem is worked

Ten stages, from the questions you would ask an interviewer to the trade-offs you would defend. Each asks one question, and the simulator runs the architecture you draw against the requirements you wrote.

01ClarificationsWhat would you ask before drawing a single box?
02Functional reqsWhat must this system actually do?
03Non-functionalWhat must it promise about speed, uptime and correctness?
04Capacity estimationHow much load and data does this have to hold?
05API designWhat does the outside world call, and what comes back?
06Data modelWhat gets stored, and what is it looked up by?
07Use-case breakdownHow does each requirement actually get served?
08High-level designWhich components handle a request, and in what order?
09Deep divesWhich part breaks first, and what do you do about it?
10Trade-offsWhat did this design cost, and what breaks at 10×?

Primary sources for this problem

Sundheim — Reliable Cron across the Planet (ACM Queue 2015, Borgcron paper)
LinkedIn — Air Traffic Controller: Member-First Notifications (2016)
LinkedIn — Hermes mass email retros (eng blog)
Pinterest — NEP Notification System and Relevance (Medium 2017)
Uber — Cherami: Uber's durable distributed task queue
Uber — Announcing Cadence (Temporal predecessor)
DoorDash — Cadence as a Fallback for Event-Driven Processing
Discord — How we store trillions of messages (Cassandra → ScyllaDB)
Stripe — Designing robust idempotency keys (Brandur)
Brandur — Implementing Stripe-like Idempotency Keys in Postgres
AWS Builders' Library — Idempotency at Scale (re:Invent ARC403 2021)
AWS — Handling SES throttling (Maximum sending rate exceeded)
AWS — SES sending quotas & dedicated IP warmup
Mailgun — Bulk email sending with queue management
Slack — Tracing notifications (Go → Kafka → Elasticsearch)
Shopify — High availability background jobs
Vallery Lancey — Kubernetes CronJob Failed For 24 Days (case study against naive cron)
Quartz Scheduler — JDBC JobStore clustering docs (the misfire-threshold story)
Cloudflare — Cron Triggers internals (Nomad-distributed schedulers)
Apache Kafka — KIP-429 cooperative incremental rebalancing
Beyer et al. — SRE Workbook ch. 21–23 (overload, cascading failures, critical state)
Campbell & Majors — Database Reliability Engineering ch. 9 (fencing tokens)

More in Queues, Pub/Sub & Event Streaming

Moving events between services without losing them: logs, work queues, fanout, change data capture, stream processing, and the delivery systems built on top.

Browse the full problem catalog, or see what the simulator does and does not model.