#38Trending Topics
Sliding windows + Count-Min Sketch + top-K.

Build the real-time Trending Topics service that powers Twitter Trends, TikTok "Trending", YouTube Trending, Reddit "Hot", Twitch "Top Streams", or Pinterest's trending pins. Engagement events stream in at scale; the system maintains the top-K most-mentioned entities over multiple sliding time windows (1 m, 5 m, 1 h, 24 h), per locale, and exposes a read API that the product surface polls every app cold-start. The algorithmic core is Count-Min Sketch (Cormode & Muthukrishnan 2005) for the long tail plus HeavyKeeper / Space-Saving (Gong 2018 / Metwally 2005) for the top-K maintainer, wired into Apache Flink's sliding-window keyed-state machinery and the Kappa architecture (Kreps 2014) for replay.

The hard part is not the sketches — it's everything around them: viral hot keys breaking partitioning, watermark stalls under regional Kafka producer lag, abuse-driven brigading inflating fake trends, UI jitter from counter-noise swaps, and the 100 ms read-SLO that has to survive a Redis shard restart. This canonical is what a staff SRE team would actually deploy: 15 components, full long-form internals, transaction-aware emit pipeline, and an on-call ergonomics story.

Reading: Cormode & Muthukrishnan — An Improved Data Stream Summary (CMS, J.Alg. 2005) · Metwally, Agrawal, El Abbadi — Space-Saving top-K (ICDT 2005) · Gong et al. — HeavyKeeper (USENIX ATC 2018) · Kulkarni et al. — Twitter Heron: Stream Processing at Scale (SIGMOD 2015) · Apache Flink — Stateful Stream Processing + Watermarks docs · Confluent — Windowing in Kafka Streams + KIP-633 grace periods · Confluent — KIP-794 Strictly Uniform Sticky Partitioner · Confluent — KIP-429 Cooperative-Sticky Rebalance · Jay Kreps — Questioning the Lambda Architecture (Kappa) · Boykin, Ritchie, Singhal — Summingbird / Algebird (Twitter) · Apache DataSketches — Theta sketch (Yahoo) · LinkedIn — Open-sourcing Apache Pinot · Discord — How Discord Stores Trillions of Messages · Datadog — March 8 2023 Multi-region outage retro · Cloudflare — June 21 2022 cross-region routing retro · Twitter — How we're fighting spam and malicious automation (2018) · Pinterest — TransAct real-time user actions for Homefeed · Beyer et al. — SRE Workbook (Managing Load + Cascading Failures)
multi-resolution sliding windows
Count-Min Sketch (Cormode-Muthukrishnan)
HeavyKeeper / Space-Saving top-K
two-stage aggregation against hot keys
watermarks + late-arriving events
Kappa replay (Kafka-as-source-of-truth)
per-locale home region
abuse-gate before promotion
hysteresis to stop top-K jitter