Trending Topics

How this problem is worked

Ten stages, from the questions you would ask an interviewer to the trade-offs you would defend. Each asks one question, and the simulator runs the architecture you draw against the requirements you wrote.

1. 01ClarificationsWhat would you ask before drawing a single box?
2. 02Functional reqsWhat must this system actually do?
3. 03Non-functionalWhat must it promise about speed, uptime and correctness?
4. 04Capacity estimationHow much load and data does this have to hold?
5. 05API designWhat does the outside world call, and what comes back?
6. 06Data modelWhat gets stored, and what is it looked up by?
7. 07Use-case breakdownHow does each requirement actually get served?
8. 08High-level designWhich components handle a request, and in what order?
9. 09Deep divesWhich part breaks first, and what do you do about it?
10. 10Trade-offsWhat did this design cost, and what breaks at 10×?

Primary sources for this problem

• Cormode & Muthukrishnan — An Improved Data Stream Summary (CMS, J.Alg. 2005)
• Metwally, Agrawal, El Abbadi — Space-Saving top-K (ICDT 2005)
• Gong et al. — HeavyKeeper (USENIX ATC 2018)
• Kulkarni et al. — Twitter Heron: Stream Processing at Scale (SIGMOD 2015)
• Apache Flink — Stateful Stream Processing + Watermarks docs
• Confluent — Windowing in Kafka Streams + KIP-633 grace periods
• Confluent — KIP-794 Strictly Uniform Sticky Partitioner
• Confluent — KIP-429 Cooperative-Sticky Rebalance
• Jay Kreps — Questioning the Lambda Architecture (Kappa)
• Boykin, Ritchie, Singhal — Summingbird / Algebird (Twitter)
• Apache DataSketches — Theta sketch (Yahoo)
• LinkedIn — Open-sourcing Apache Pinot
• Discord — How Discord Stores Trillions of Messages
• Datadog — March 8 2023 Multi-region outage retro
• Cloudflare — June 21 2022 cross-region routing retro
• Twitter — How we're fighting spam and malicious automation (2018)
• Pinterest — TransAct real-time user actions for Homefeed
• Beyer et al. — SRE Workbook (Managing Load + Cascading Failures)