View Count on a Video/Post — a worked solution

Typical clarifications to surface:

• What is a "view"? A page load, or watch-time past a threshold? Drives the client gate and the dedup window. We assume watch-time-gated.
• Unique viewers or total views? Both are usually wanted; uniques are approximate (HyperLogLog). Drives the dedup store design.
• How exact must the public count be? Approximate + eventually consistent (seconds of lag) is universally accepted for a display count; advertiser/creator-paid counts need a verified, auditable number on a slower cadence.
• Must the count be monotonic? Users hate counts going down — but fraud purges legitimately lower it. We serve monotonic for speed-layer wobble and allow the verified layer to correct.
• Retention? Raw events: weeks (then drop / cold). Aggregates: forever. Legal/PII constraints on raw viewer data.
• Multi-region? Counts are commutative, so active-active is natural — unlike a mutable balance.

Assumptions to state:

• 5B view events/day (~58k/s avg, ~174k/s diurnal peak, +100k/s on a single viral video).
• Count reads ≈ 8× view events → 1.5–2.5M reads/s, almost all CDN-absorbable.
• A view = watch-time past a threshold, deduped per (viewer, video) over a 24–48h window.

• Record a view: accept a fire-and-forget beacon, dedup it, and durably log an idempotent event.
• Display the total count: serve a tiny, cacheable, approximate count fast and globally.
• Count unique viewers (approximate, via HyperLogLog).
• Filter invalid traffic: cheap inline GIVT at ingest; sophisticated SIVT in a batch audit that produces a verified count.
• Creator analytics: views over time by geo / device / source for a dashboard.
• Reconcile: a batch layer recomputes the authoritative count from the durable event log and corrects the serving store.

SLOs:

Error budget. Read availability 99.99% ≈ 52 min/yr. But the failures that actually govern on-call here are not unavailability — they are wrong numbers: count inflation (dedup failed), count staleness (consumer lag), count going backwards (checkpoint replay), and silent undercount (dropped beacons). The headline burn-rate alerts are the count-staleness SLO (display lag past 60s) and a monotonicity alarm (any served public count that decreases for a non-verified reason pages immediately). But those two — and the inflation canary (ingested ÷ unique-estimate) — are all computed FROM the pipeline that would be wrong, so the load-bearing signal is the independent synthetic canary: a prober that writes K known views through the real front door and reads the counter back out-of-band, paging if the count didn't move by K within SLO. It shares no failure domain with the pipeline, so it catches the silent gray failure (counts wrong while every endpoint returns 200) that the derived SLIs can miss. Multi-window multi-burn-rate per the SRE Workbook.

Anchor on a YouTube/TikTok-tier platform and derive every shard/partition/cell number rather than guessing.

1. Throughput. YouTube serves ~1B watch-hours/day ≈ 5B views/day; TikTok ~1B/day. Take V = 5e9 views/day. ingest avg = 5e9 / 86,400 ≈ 57,900 view-events/s. Diurnal peak ×3 ≈ 174k/s; a single viral video adds +100k/s on top → design for ~250k view-events/s ingest. Count reads ≈ 8× ingest → ~1.4M/s at diurnal peak, ~1.5–2.5M/s once viral-read headroom is folded in (a trending video is re-read far more than 8× its view rate), ~99% absorbed by the edge (counts are tiny, public, and stale-tolerant).

2. Storage asymmetry — the whole point. raw event ≈ 100 B (video_id, viewer/session, ts, geo/device, event_id) → 5e9 × 100 B = 500 GB/day (~3.5 TB/week, ~180 TB/yr if kept). vs. aggregated counter ≈ 24 B × 1e9 videos ≈ 24 GB total, kept forever. Raw events are huge and ephemeral (30-day hot in the lake, then cold/expire); the aggregates are tiny and permanent. This asymmetry is why raw events go to cheap object storage and counters live in a small, fast store.

3. The hot key — the central scaling problem. Views are power-law: one viral video can take 100k views/s — i.e. one counter key taking 100k writes/s.

• A single Redis node does ~100k–180k INCR/s; a single DB row serializes on its lock at ~1,000 writes/s before it falls over.
• Fix = sharded counter: split one logical counter into N cells, write key:{rand(N)}, read = SUM(cells). For a DB-row-class store, counterShardsHotKey = 100,000 / 1,000 = 100 cells; read amplification = N reads per displayed count (the trade).
• And pre-aggregation: the Flink window collapses 100k events/s into one batched UPSERT per (video, region) per window, so the counter sees ~1 write/window, not 100k/s.

4. Log partitions. streamPartitionsMin = 174k/s ÷ ~1k events/s/partition ≈ 174, rounded up to 256 (next power of two, for headroom + clean rebalancing). A viral video still maps to one partition (hot), mitigated by key-salting the hottest keys.

5. Dedup / uniques state. Exact unique-set ≈ 40–60 MB for a 1M-viewer video in a Redis set (per-member overhead dominates; the raw 16-byte ids alone are 16 MB) — and it grows without bound. HyperLogLog = 12 KB/video at ~0.81% error, regardless of cardinality. Active working set ≈ 10M videos × 12 KB ≈ 120 GB sharded Redis; sketches age out with a 24–48h window.

6. Where the architecture flips.

Record a view (fire-and-forget beacon):

POST /v1/beacon
Content-Type: application/json
{ "video_id": "vid_123", "event_id": "9f1c-...", "event_time": 1717795200, "session": "s_88", "ms_watched": 7400 }

204 No Content        # acked immediately; dedup + log happen before ack, aggregation is async

event_id + event_time are the idempotency key: a retried or double-fired beacon is deduped, not counted (Netflix's counter-event design). ms_watched is re-checked server-side against the watch-time threshold.

Read the display count (cacheable):

GET /v1/videos/vid_123/count
200 OK
Cache-Control: public, max-age=30, stale-while-revalidate=30
{ "video_id": "vid_123", "views": 1048217, "as_of": "2026-06-08T12:00:30Z", "verified": false }

verified:false is the fast speed-layer number; the nightly audit replaces it with verified:true (which may be lower). The public count is views only — the unique-viewer estimate is not on this path. The HLL sketch lives in the dedup Redis (off the read path) and as a Theta sketch in the analytics OLAP store; uniques are served only by the authed Studio endpoint below, which reads that OLAP store. Putting unique_estimate here would require the aggregator to also project a merged per-video sketch into the count cache / counter store — see Open questions.

Creator analytics (authed, not cacheable):

GET /v1/studio/vid_123/views?from=...&to=...&by=country,device   (Bearer)
200 OK   { "series": [ ... per-bucket rollups ... ] }
403 Forbidden   # bearer principal does not own / is not authorized for vid_123

The bearer token is authentication; the read service additionally runs a per-resource authorization check — the principal must own (or be granted access to) video_id — before serving, and returns 403 otherwise. Without it any authenticated creator could read any other creator's private, competitively-sensitive analytics. The principal→video ownership lookup is part of the analytics read flow (see # hld Security).

View event (immutable, in the log + lake): video_id, event_id (idempotency), event_time, viewer/session, region, coarse device/geo/source, ms_watched. The event_id makes the whole pipeline replayable and dedupable.

Dedup state (Redis, TTL'd): seen:{viewer}:{video} and seen:{event_id} keys (24–48h TTL); hll:{video} HyperLogLog sketch for uniques. Bounded, reconstructable, noeviction.

Counter cells (Cassandra/Scylla): (video_id, region, shard) → rolled_up_count written as idempotent windowed UPSERTs; total = SUM. A separate verified_count column the batch layer overwrites.

Analytics rollups (ClickHouse/Druid/Pinot): (date, video_id, country, device, ...) → views, theta_sketch(uniques) materialized at merge time.

Database choice — recommended. Kafka for the event log, Redis for dedup + HLL + the count cache, Cassandra/Scylla (idempotent rollup cells, not counter columns) for the durable serving counter, ClickHouse/Druid/Pinot for analytics, S3 for the raw-event lake. The non-negotiable: the hot path appends an idempotent event to a durable log and the count is a projection aggregated asynchronously — never a synchronous non-idempotent increment against a strongly-consistent store.

Ingest / write path. Client → (CDN pass-through) → Gateway → View Ingest Service → Dedup Store, then Ingest → View Event Log. The gateway sheds GIVT and rate-limits; ingest dedups (SETNX + event_id), updates the HLL, and publishes an idempotent event to Kafka. The client gets a 204 immediately — nothing on this path increments a counter.

Read path (the product, ~99% edge-served). Client → CDN. On an edge miss: → Gateway → Count Read Service → Count Cache; on a cache miss the read service single-flights a SUM of the video's counter cells and back-fills the cache. The read service applies the monotonic guard.

Speed layer (the hot-key absorber). View Event Log → Stream Aggregator (Flink). Per video, a tumbling window collapses N events into ONE batched, idempotent UPSERT into the Counter Store, a versioned SET into the Count Cache, and a rollup into the Analytics OLAP store. This is what lets a 100k-views/s video survive on one logical counter.

Batch / verified layer (the YouTube 301). View Event Log → Raw Event Lake (S3 sink). On a schedule, the Verified-Views Auditor scans the lake, enriches against the external IVT Signal Feed (circuit-broken, off the hot path), re-scores sophisticated fraud, and writes the authoritative verified count back to the counter store and analytics — the one sanctioned source of a count decrease. A DLQ parks poison events so one bad message can't stall a partition.

Independent watchdog. The Synthetic View Canary sits outside the pipeline entirely: it writes K known probe views through the real front door and reads the counter back directly (bypassing the cache and CDN), paging if the count didn't move by K within the freshness SLO. Because every other "is the count wrong" signal is derived from the pipeline that would be failing, this out-of-band prober is the only one with independent ground truth — the dead-man's-switch for the silent gray failure where counts are wrong but every endpoint returns 200.

Why two numbers. The speed layer gives a fast, approximate count (verified:false); the batch layer gives a slow, authoritative one (verified:true). Lambda architecture: the batch layer reconciles what the speed layer approximated (Twitter TSAR/Summingbird, Cloudflare's ClickHouse rollups, Netflix's interval rollup).

The hot key (one viral video)

A single video at 100k views/s is 100k writes to one counter key. A DB row dies at ~1k/s; a Redis node at ~100k/s. Four mechanisms stack: (1) the log absorbs the spike and decouples writes from aggregation; (2) window pre-aggregation collapses N events → 1 UPSERT/window; (3) sharded counter cells spread even that one write across N cells (read = SUM); (4) key-salting the hottest log partitions so one video doesn't pin one partition. The edge absorbs the read side entirely.

Idempotency & the counter-column trap

The stream is at-least-once and Flink's exactly-once is internal-state only — an external counter write re-fires on checkpoint recovery. So the external write must be idempotent: a (video, region, window_id)-keyed UPSERT, not an INCR. Cassandra counter columns are the classic trap — non-idempotent, so a timed-out retry overcounts and a dropped one undercounts; "not even eventually consistent." Ably's post-mortem: a client retry loop on counter columns drove every node to 100% CPU across all regions, recurring at 24h intervals, until they removed counter columns entirely. The dedup store (event_id) is the upstream defense; idempotent UPSERTs are the downstream one. And the idempotency hinges on window_id being derived from event-time, not wall-clock, so a checkpoint replay re-derives the identical key instead of re-bucketing a late event into a new window and double-counting.

The txn.group "view-rollup" (mode event-sourced) tags the two edges that define the count's correctness — ingest→log (participant event) and aggregator→counter (participant projection) — because the log is the system of record and the counter is its replayable projection. This is event-sourced, not outbox: ingest has no local DB write to make atomic with the publish (the dedup SETNX is best-effort), so there is nothing for an outbox to atomically commit. The raw-event sink (log→lake) and the OLAP rollup (aggregator→analytics) are also projections of the same log, but are intentionally left out of the group so it stays focused on the serving-count correctness contract.

Why the count goes backwards (two causes)

Bug: a Flink checkpoint replays an older state → the emitted count decreases. Fix: the read tier's monotonic guard serves max(value, last-served) for the speed-layer path so a regression never reaches users. By design: the batch verifier purges confirmed-fake views → the verified count is legitimately lower. This is YouTube's 301+ freeze productized: the count froze pending a bot audit, then jumped — or dropped — when fakes were purged (now continuous AI validation + daily batch audits). The serving tier distinguishes them: speed-layer regressions are clamped; a verified write is the one sanctioned decrease.

How the verified decrease survives the guard AND the speed layer (the load-bearing detail). Two traps the mechanism must avoid, and both must be handled or "counts go down" silently doesn't work. (1) The monotonic guard could clamp the legitimate decrease back up — so every count value carries a verified bit; the guard exempts a verified value from the max() floor AND resets last_served down to it, so subsequent speed-layer reads clamp to the post-purge floor, not the pre-purge number. (2) The next speed-layer window could re-add the purged views — so the verifier writes a SEPARATE authoritative column the read-SUM prefers over the speed-layer cells, and advances the speed layer's event-time watermark past the audited day so the aggregator can't re-emit those events. Without (1) the verifier is a silent no-op (the count never drops); without (2) the correction is erased on the next ~10s tick. The verified:false/true bit already on the API response is exactly this stamp, threaded from the verifier through the counter column into the read tier's guard.

Dedup, uniques, and silent inflation

Dedup is a TTL'd SETNX per (viewer, video) + event_id; uniques are a HyperLogLog (12 KB/video, 0.81% error vs 400–600 MB for an exact set — Reddit's design). The killer failure is silent: if the dedup Redis uses LRU and hits maxmemory, it quietly evicts dedup keys → already-counted views count again → fleet-wide inflation exactly during peak. The fix is a one-liner: noeviction + page on ANY eviction (fail loud), and keep the dedup Redis physically separate from the count cache so cache churn can't evict dedup state.

Approximate vs exact, speed vs batch

The display total is approximate (summed cells, seconds stale) and that is fine — nobody needs the millionth view to be exact in real time. Uniques are approximate by construction (HLL/Theta). The verified count is exact-ish and slow. Choosing what may be approximate is the core latency/cost lever: it is what lets the hot path be a fire-and-forget append.

Observability

SLIs (RED/USE): Ingest — beacon accept/reject, publish error, ingest→log lag (RED). Log — produce/consume rate, per-partition consumer-lag VARIANCE (one viral partition lagging is the signal, not aggregate lag). Aggregator — checkpoint duration/failures, watermark lag, window backlog. Counter — per-shard CPU/latency skew (hot-key detector), read amplification. Dedup — evicted_keys (page on any), memory %, dedup-miss trend. The headline alarms: (1) count-staleness (display lag > 60s); (2) monotonicity (a served public count decreased for a non-verified reason — pages immediately); (3) inflation ratio (ingested-events ÷ unique-estimate above baseline). But (1)–(3) are all computed FROM the pipeline that would be failing — a confidently-wrong or smoothly-inflating pipeline keeps them green, and the inflation ratio divides two outputs of the same dedup/HLL pipeline so a shared bug can move both together — so the load-bearing alarm is (4) the independent synthetic canary: K known probe views written through the real front door, the counter read back out-of-band (bypassing cache + CDN), paging if it didn't move by K within SLO. It is the only signal with ground truth independent of the pipeline, paired with a 'canary-not-reporting' alarm so a dead canary can't hide behind a flatlined-green SLI. Dashboards on-call open: display-count staleness by region; per-partition log lag (variance); counter hot-shard skew; dedup memory/eviction; verified-vs-raw divergence + verifier audit lag; DLQ depth/age; canary probe-delta + liveness (the independent signal). Tracing: the ingest path is head-sampled (~1%); the verified-recount path and the canary's write-then-read are 100% sampled so a wrong correction or a missed probe always has a trace.

Deploy & runtime

Aggregator deploys use cooperative-sticky + static consumer-group membership so a rollout doesn't trigger a rebalance storm (each rebalance pauses consumption and feeds lag); externalized/verified checkpoints so a deploy can't replay a stale state into a regression. Counter store: rolling, in-shard followers so a node restart doesn't drop a cell. Count cache: rolling restart + single-flight so a cold cache doesn't stampede the counter store. Schema: the view-event schema is registry-managed with BACKWARD compatibility, rolled out consumer-first; incompatible events hit the DLQ, not a crash-loop. Secrets: the IVT feed's API key and inter-service mTLS certs auto-rotate with expiry-burn alarms.

Multi-region / DR

Posture: active-active. Counts are commutative — each region ingests views locally, appends to its regional log, aggregates into its own (video, region) counter cells, and serves reads locally; the global count is the SUM of per-region cells, merged async. There is no single-writer bottleneck and nothing to reconcile on a partition (unlike a mutable balance) — the deepest reason view counting is multi-region-friendly. DR: losing a region loses only that region's in-flight, un-replicated increments (RPO ≈ the cross-region merge lag, seconds) and its share of ingest until traffic reroutes; other regions' cells are unaffected and the global SUM simply omits the dead region's most recent delta until it recovers and replays from its durable log. Be honest about the two RTOs: the counter store's leaderless quorum gives RTO ≈ 0 for node loss, but losing a whole region's ingest tier has an RTO of minutes — the traffic-reroute window bounded by DNS TTLs, resolver caching, and sticky clients, during which that region's views are simply dropped (best-effort → a bounded undercount, recoverable at the next verified audit from the durable log) and its un-merged cells are absent from the global SUM until it recovers. Claiming a single sub-second number here is how a postmortem ends up explaining why the count "lagged for 8 minutes." The caveat on scope: this canvas draws ONE region's ingest+counter tier and bakes the active-active posture into the counter store's internals (leaderless, (video, region, shard) partitioning, async cross-region merge) rather than duplicating every node per region — see Open questions.

Security

AuthN/Z: beacons and public count reads are anonymous (auth:none, edge-served); Studio reads require a bearer token AND a per-resource authorization check — the read service resolves the bearer principal's identity, looks up ownership/ACL for the requested video_id, and serves only if the principal owns or is granted that video, else 403. Authentication alone is not enough: without the ownership check any authenticated creator could read any other creator's private analytics. Every internal hop is mTLS; the IVT feed uses an API key. PII: raw events carry viewer/geo signals, so the lake is encrypted (SSE-KMS), access-controlled, and lifecycle-expired — aggregates (the only thing kept forever) carry no per-viewer data. Threat headlines: (1) view fraud / inflation — bot farms inflating a count for ranking or ad revenue; mitigated by GIVT at the edge, watch-time gating, per-device rate limits, and the SIVT batch audit; (2) beacon spoofing — forged beacons; mitigated by rate limits + the dedup/idempotency key + batch re-scoring; (3) IVT-feed compromise / bad verdicts — mitigated by shadow-scoring before enforcing and a manual-review queue.

Trade-offs we accepted (and the alternative we rejected)

• Fire-and-forget event log + async rollup over synchronous count-on-write. Rejected sync increment because it dies at ~10k/s and makes the hot path a non-idempotent write to a consistent store. Cost: the count is eventually consistent (seconds of lag).
• Idempotent windowed UPSERTs over Cassandra counter columns. Rejected counter columns as a documented production trap (Ably). Cost: the aggregator must carry window_id keys and dedup logic.
• Sharded counter cells (read = SUM) over a single counter row. Rejected the single row because a viral key serializes on one lock. Cost: read amplification (N cell reads per display).
• Two numbers (fast approximate + slow verified) over one exact real-time count. Rejected exact-real-time because fraud detection can't run on the hot path. Cost: raw and verified diverge until the audit, and the public count can legitimately drop.
• HyperLogLog uniques over exact sets. Rejected exact sets (400–600 MB/hot video). Cost: ~0.81% error on uniques.
• noeviction dedup Redis over LRU. Rejected LRU because silent eviction = silent inflation. Cost: must size for peak + page on eviction (fail loud, not graceful).
• Active-active commutative counters over single-master. Rejected single-master because counts don't need it and it adds cross-region write latency. Cost: the global count is a sum that's briefly behind by the merge lag.
• Fail-open ingest with downstream event_id dedup over fail-closed. Rejected fail-closed because dropping views is a silent, unrecoverable undercount on a paid number; rejected naive fail-open because counting blind manufactures the inflation-then-purge drop. Cost: the per-(viewer,video) suppression leaks during a dedup outage (a re-view may count), bounded and corrected by the audit.
• An independent out-of-band canary over relying on pipeline-derived SLIs. Rejected derived-only monitoring because every "count is wrong" signal computed from the pipeline goes green when the pipeline is confidently wrong. Cost: a separate prober plus its own liveness watchdog to run.
• R=1 quorum reads on the counter over R=2. Rejected R=2 because a SUM over many cells is approximate-by-construction (not an atomic snapshot) and the count is eventual anyway, so per-cell read quorum buys nothing. Cost: a single lagging replica can briefly skew one cell's contribution — invisible against the seconds-of-staleness the display already tolerates.

Failure modes & mitigations

Open questions for human review

• Single-region canvas, active-active posture. We bake active-active into the counter store's internals rather than drawing a second region's ingest+counter tier (the teaching focus is dedup/bot/aggregation, not cross-region staleness). Confirm that altitude vs a fully duplicated multi-region drawing.
• Counter store = Cassandra rollup cells vs DynamoDB vs a custom counter (Netflix-style TimeSeries + EVCache). All three are real; we chose idempotent Cassandra cells. A reviewer who wants Netflix's exact event-store-+-rollup-+-EVCache shape would add a TimeSeries node.
• Realtime OLAP ingest. We route analytics through the aggregator (pre-aggregated rollups). Pinot/Druid commonly ingest from Kafka directly; that would be a stream → analytics edge instead of (or in addition to) aggregator → analytics.
• GIVT depth at the gateway. We keep inline filtering to signature/heuristic (no per-view external RPC). Confirm that's the right line vs a lightweight in-region bot-scoring sidecar for a subset of high-value beacons.
• Verified-count UX. Showing a count that can drop is a product decision. Some platforms hide the drop (serve monotonic and silently correct internal/ad numbers); confirm whether the public number is allowed to decrease.
• Dedup fail-open vs fail-closed. On a dedup-store outage we fail OPEN (the per-(viewer,video) suppression leaks) backstopped by the aggregator's in-window event_id dedup and the daily verifier; a team that treats any inflation as unacceptable might fail CLOSED (drop views during the outage) instead. Confirm the posture for this product.