YouTube / Netflix Streaming — a worked solution

What each component is for

Client

Three concrete callers share this node: (a) the playback player (Shaka / hls.js / ExoPlayer / AVPlayer / dash.js) that fetches manifests + segments + DRM licences; (b) the upload UI used by creators to push mezzanine sources; (c) telemetry on the same player emitting playback events back to the popularity pipeline. The player runs an ABR algorithm (BBA, BOLA, or Pensieve-class) that picks rungs from the manifest based on buffer level and throughput estimate.

Why it exists. Considered modelling the upload UI as a separate edge node because its failure modes (chunked-resumable upload, large body, retryable PUT) differ from playback. Rejected because both run on the same SDK and ship from the same release pipeline — splitting them only matters when we model authn (OAuth + device binding for playback, signed-upload-key for creators), and that lives at the gateway downstream. Player IS the surface where the ABR regression that puts the fleet on 480p actually originates.

When it fails. Bad client deploy regresses ABR — within minutes, fleet drops to lowest rung; egress goes DOWN, error rate stays zero, p99 latency looks fine. Detection: average_delivered_bitrate, rebuffer_ratio, rung_switch_rate. Mitigation: 1% / 10% / 50% staged rollout gated on bitrate-delta SLO. SDK quickstarts ship the canonical ABR + retry helpers so naive clients don't regenerate manifest URLs on retry.

CDN · Open Connect EdgeNetflix Open Connect (FreeBSD + NGINX) / CloudFront / Akamai

Serves manifests + segments to nearby viewers. One appliance is 96 GB RAM + 16 TB NVMe + dual 100 Gb NIC, sustains ~90 Gb/s TLS at ~55% CPU under Netflix's kernel-TLS + sendfile + RACK/BBR build. Holds the ~5-10% of catalogue popular in this country (per-region popularity feed). Serves manifests with short TTL (60-300 s VOD, 1-5 s live), serves segments with long TTL (hours-to-days, immutable URL).

Why it exists. Considered using only a public CDN (Cloudfront / Akamai) — that's how Disney+, Hotstar, and most others run. Rejected for Netflix-class scale because at >5% of any ISP's traffic, settlement-free peering economics flip in favour of embedding appliances inside the ISP's rack: no transit costs, predictable latency, ISP gets a free peering agreement. Open Connect carries ~100% of Netflix bytes; the public CDN model is the right answer for catalogues that haven't crossed the embed threshold.

When it fails. Most-watched failure: BGP withdrawal / mis-route at the CDN provider (Fastly 2021-06-08 pattern) makes the whole edge tier unreachable from the public internet — internal health checks pass because internal paths are fine. Detection: external synthetic monitor 5 s polling against a non-CDN'd path, SYN-ACK rate by ASN. Mitigation: multi-CDN steering with real-user-measurement (Conviva / NPAW). Hot-key on a viral release saturates one OCA's outbound link; mitigation is per-segment sharding + edge prefill of top-N titles.

CDN · Regional ShieldOpen Connect storage tier / CloudFront Origin Shield / Akamai tiered cache

Sits between the edge POPs and the origin. When the edge OCA misses, the request falls through here before touching the object store. Sized at ~10x edge capacity per region; holds the long-tail catalogue that doesn't fit in edge NVMe. Akamai calls this 'tiered cache', CloudFront calls it 'Origin Shield', Open Connect calls it the 'storage tier'.

Why it exists. Considered going edge → origin direct (no shield). Rejected because the cache miss explosion at origin is the main cost driver — without the shield, origin egress is ~1/cacheHitRatio of total egress; with a shield at 95% hit, origin sees ~0.05 × 0.05 = 0.25% of traffic instead of 5%. The shield exists to protect the origin's bandwidth bill and the metadata path, not to lower viewer latency.

When it fails. Shield outage pushes 100% of edge misses to origin — that's a ~20x origin spike (5% miss rate × 1/0.05 amplification). Detection: shield_hit_ratio, origin_5xx_rate, origin_egress_gbps. Mitigation: edge falls through directly to origin with a stale-while-revalidate budget on the popular tier; long-tail viewers see latency tail. Cross-region replication of the shield's catalogue prefill so a regional shield outage doesn't take a full region's mid-tier offline.

API GatewayEnvoy + Cloudflare WAF + TLS terminator

L7 ingress for every request that misses the edge: manifest builds, license fetches, uploads, telemetry, control-plane. Terminates TLS, runs the WAF (token-auth signature check, country geo-block, rate-limit per device), copies the entitlement claim from the JWT onto upstream headers signed with HMAC, mints short-lived signed URLs the manifest service injects into the response.

Why it exists. Considered baking auth + signed-URL minting into the manifest service. Rejected because (1) license requests + upload requests need the same edge concerns, so duplicating them is bug-prone, (2) the WAF rule set (geo-blocks, abuse signatures) belongs at one layer that fronts everything, (3) signed-URL minting holds an HMAC secret that should NOT live next to product code. Centralizing auth means upstream services trust the signed envelope without round-tripping.

When it fails. Gateway pool exhaustion during a live-event join surge masks as 'manifest fetch slow' but is actually upstream backpressure from license / manifest. Detection: gateway_active_streams, downstream_5xx_rate. Mitigation: drain mode + per-zone failover; circuit-break a misbehaving device fingerprint before 503'ing the world. The Disney+ 2019-11-12 launch lesson: test at 5x expected peak, not at average — premiere nights hide a 20x minute-burst inside a 5x hour-average.

Service · Manifest BuilderGo / Java; HLS m3u8 + DASH mpd + CMAF

Generates the master + media manifests on demand. Reads the asset's encoded variants (rungs × codecs × DRM systems) from the catalogue cache; if missed, falls through to the metadata DB. Injects signed URLs (minted by the gateway) for every segment, applies the per-region geo-rule (rung trimming for poor networks, codec gating per device class), and writes the rendered manifest back into the edge with a 60-300 s TTL.

Why it exists. Considered serving manifests directly from object storage as static files. Rejected because (1) signed-URL injection requires a per-request HMAC signature that must NOT live in S3, (2) per-device rung trimming (don't expose AV1 to a 2019 iPad) is dynamic, (3) the manifest re-renders every TTL anyway. Considered building manifests at the edge — same problem with HMAC custody. Centralized rendering is the simplest design that keeps signing keys away from caches.

When it fails. Catalogue-cache cold start: hit ratio drops from 95% to 60% during a regional cache restart, manifest p99 climbs from 30 ms to 250 ms because every miss touches the DB. Detection: catalogue_cache_hit_ratio, manifest_build_p99. Mitigation: warm the cache from the popularity feed BEFORE the restart; bounded-staleness manifests (serve last-known-good manifest with a longer TTL during cache warmup). The metadata-DB primary failure mode is what the trace simulator's manifest-build-cold-db trace exists to model.

Auth / DRM License ServerWidevine (Google) + FairPlay (Apple) + PlayReady (Microsoft)

Two concerns under one roof: (1) entitlement — does this account have rights to play this title in this country today? (2) DRM license — return a content-key wrapped to the device so the player can decrypt the stream. Validates the signed envelope from the gateway, checks entitlement against metadata DB (with a hot Redis cache), and returns a Widevine / FairPlay / PlayReady license depending on what the device asked for. Widevine ≈ 60-65% of devices, FairPlay ≈ 25-30%, PlayReady fills the rest.

Why it exists. Considered splitting entitlement and license minting into two services. Rejected because (1) every license fetch needs the same entitlement check (skip it and you've broken DRM contracts), (2) the latency budget is tight (p99 ≤ 200 ms from the player) and an extra hop breaks it, (3) the failure modes are coupled: entitlement DB down → license fetch fails. We do split the license server pool by DRM system internally (different binary deploys for Widevine vs FairPlay so a Widevine bug can't tank FairPlay), but they share the same entitlement code path.

When it fails. Premiere-night overload: every player on Earth requests a license in the same 5-min window; pool saturates, exponential back-off retries make it worse. Disney+ Mandalorian 2019-11-12 was exactly this. Detection: license_5xx_rate, license_fetch_p99, gateway_active_streams. Mitigation: shadow-load test at 3x peak, license-server pre-warm, regional sharding, pre-authenticated tokens. The 'license server overload' chaos card models this directly.

Cache · Catalogue (EVCache)Netflix EVCache (Memcached fork, Ketama consistent hashing)

Stores asset:{titleId}:variants (rungs × codecs × DRM × signed-URL templates), entitlement:{accountId}:{titleId} (rights-cache for license server), popularity:{titleId} (per-region rank). Cache-aside on read: manifest service tries here first (p99 ~5 ms), falls through to DB on miss; DRM service tries here first too. Populated lazily on miss; INVALIDATED on every catalogue write so a buggy writer can't poison the cache.

Why it exists. Considered Redis cluster (we already use it elsewhere). Picked EVCache (Memcached + Ketama) because the read pattern is 99.9% point-lookup with no need for set/list/sorted-set ops, and EVCache's heterogeneous-cluster allocation means losing one shard takes 1/N of the keyspace offline (graceful degradation), not a quorum failure. Considered making the cache the source of truth for popularity scores. Rejected because the popularity worker recomputes them from the playback stream — making the cache authoritative would mean every popularity update has to invalidate it anyway.

When it fails. Cache cold restart sends all manifest builds to the DB → manifest p99 climbs from 30 ms to 250 ms; DB connection pool saturates; manifest builds start 503'ing. Detection: cache_hit_ratio, manifest_p99, db_pool_active. Mitigation: warm from the popularity feed BEFORE restart; single-flight on the manifest service so 1000 concurrent misses for the same key don't hammer the DB 1000 times. Hot shard on a viral release: kill-shard chaos card models this directly.

SQL DB · Metadata (Vitess)Vitess (sharded MySQL) — YouTube pattern; alt: Spanner

Source-of-truth for the catalogue: title rows (id, region rights, takedown state), asset rows (variants × codecs × DRM × URLs), entitlement rows (accountId × titleId × geo × expiry), popularity scores. Schema is sharded by titleId (Vitess vindex on title_id_v) for a 64-shard layout. Reads served by semi-sync followers within the AZ; writes go to the shard primary; cross-shard joins are forbidden — denormalize at write time.

Why it exists. Considered DynamoDB (single-key access, zero ops). Rejected because (1) takedown queries — 'show me everything in country X marked disabled this week' — need indexes that don't fit DynamoDB's GSI model cleanly, (2) the auditor escape-hatch matters: regulators ask for 'show me all titles licensed by Disney with rights in Brazil' and you need ad-hoc SQL. Considered Spanner — it's the right answer for global serializable, but at our QPS the cost is ~3-5x Vitess for the marginal consistency we don't need (catalogue writes are not contended).

When it fails. Primary failover during prime time: 30 s of write outage on one shard; reads from followers continue at bounded staleness. Manifest builds that hit catalogue cache (95% path) survive cleanly; manifest builds that miss BREAK for the failover gap. New title launches that race the gap surface as user-visible publish delays. Detection: shard_leader_health, oldest_unreplicated_lsn, manifest_build_cold_p99. Mitigation: catalogue cache absorbs 95% of reads through the gap; new launches gated on synthetic 'first-watch' probe.

Object Store · Hot (S3 Standard)AWS S3 Standard / GCS / Colossus, multi-region replicated

Stores everything that can be read by the CDN: source mezzanines (ProRes / IMF, ~400 GB/hr), the full encoded ladder (10-50x source, ~7-10 rungs × 3 codecs × 2-3 audio tracks per asset), packaged CMAF + DRM-encrypted segments, manifests-on-disk for offline rebuild. Sized for the active catalogue (single-digit PB at Netflix-class, ~2 EB at YouTube-class). Replicated cross-region asynchronously (RPO minutes-to-hours).

Why it exists. Considered self-hosting on hardware (Open Connect appliances ARE Netflix-built hardware, but for serving — not for the origin tier). Rejected because the origin needs 11×9s durability across multiple AZs and the operational burden of running tape-as-storage at this scale is enormous. S3 + cross-region replication gives us the durability story; we layer caching (OCAs + regional shield) on top to dodge the egress bill.

When it fails. us-east-1 partial outage (2017-02-28, 2021-12-07): uploads fail, encoder writes fail, cold-tail viewers fail, hot bytes keep playing because OCAs are the source of truth for those. Detection: PutObject_5xx_rate, transcode_publish_lag, oldest_unreplicated_master_age. Mitigation: warm-warm cross-region; route encoder writes to secondary region during the outage; do NOT flip read traffic (OCAs already serve). The 's3-origin-region-outage' chaos card models this.

Object Store · Cold ArchiveS3 Glacier Deep Archive / GCS Coldline / tape

Holds the ~95% of mezzanine masters and encoded variants that aren't actively served — long-tail catalogue, retired-but-licensed titles, the 50-year retention horizon on every master. Retrieval is batched (hours) and cheap; storage is ~$1/TB/month for Deep Archive vs ~$23/TB/month for Standard. Tier-mover (the popularity worker) demotes here when popularity drops; promotes back to hot when a back-catalogue title trends.

Why it exists. Considered keeping everything in S3 Standard. Rejected on cost: at YouTube-class catalogue (~2 EB), the standard-tier cost differential is in the hundreds of millions per year, and the hit ratio on the cold archive is by definition <0.1%/asset/month. Considered tape-only (the way old broadcasters do it). Rejected because retrieval latency (days) is too long for popularity surges — we need 'hours, not days' to move a trending title back to hot.

When it fails. Glacier retrieval rate-limit: a popularity surge that needs 1000+ titles restored simultaneously hits the per-account rate limit; restore queue grows to days. Detection: glacier_restore_queue_age, popularity_promote_lag. Mitigation: pre-emptive restore of trend-candidates 24 h ahead via the popularity worker's prediction model; tier-promote scheduling that batches restore requests.

Service · Upload OrchestratorCosmos ingest control-plane; S3 multipart-upload pre-signed URLs

Three control-plane endpoints — bytes do NOT flow through this service. (1) POST /v1/uploads/init: validates creator ownership against metadata-db, validates declared content-type / size, creates an S3 multipart-upload session, returns the upload-id + a set of pre-signed PUT URLs (one per part). (2) Client uploads each part DIRECTLY to S3 against those URLs. (3) POST /v1/uploads/:id/complete: receives the per-part ETags from the client, calls S3 CompleteMultipartUpload (a metadata-only API), validates the resulting object's checksum + size, then writes the asset row to metadata-db and publishes the encode-job to Kafka in the same outbox txn. The orchestrator stays out of the byte path entirely.

Why it exists. Considered routing bytes through the service (the URL-shortener writesvc shape). Rejected because at YouTube-class ingest of ~3.3 GB/s sustained, pass-through doubles the bandwidth bill (client→service + service→S3 = ~6.6 GB/s of provisioned egress) AND the service tier becomes the bandwidth bottleneck — a 30-replica pool simply cannot saturate 6.6 GB/s without becoming an absurd network appliance. Considered fronting S3 with a transparent proxy that streams chunks. Rejected because the proxy still sits in the byte path with the same economics — direct-to-S3 with signed URLs is what every reference design (Cosmos / MezzFS / Disney+ ingest / Hotstar) actually does. The orchestrator's job is auth + lifecycle + transactional publish, not byte forwarding.

When it fails. Orphan multipart uploads: client crashes between init and complete; bytes accumulate in S3 with no asset row referencing them. Detection: incomplete_multipart_uploads_age, S3 storage-class growth rate. Mitigation: S3 lifecycle policy auto-aborts after 7 days; nightly sweeper reconciles open multipart-uploads against issued init records. Signed-URL leak: a creator's URL is exfiltrated and used by an attacker to upload garbage to the slot. Detection: per-uploadId byte-hash + per-init session-token. Mitigation: 15 min URL TTL + per-IP rate limit on init. Orchestrator down: new uploads can't START, but in-flight uploads continue (clients have signed URLs and S3 is up) — graceful degradation. Detection: upload_init_5xx_rate, signed_url_mint_failures.

Stream · Transcode BusKafka 3.7 (3 topics: encode-jobs, chunk-ready, playback-events)

Three logical topics on one cluster: (1) encode-jobs — upload publishes, encoder consumes; ~250 K work items per 30-min episode (rungs × codecs × ~900 shots) at full per-shot fan-out. (2) chunk-ready — encoder publishes when an encode chunk is durable, packager consumes for stitching + DRM-encrypt. (3) playback-events — CDN edge emits view events, popularity worker consumes for tier scoring. Single cluster keeps cross-topic ordering possible when needed.

Why it exists. Considered separate clusters per topic (better blast-radius isolation). Rejected because the operational overhead of running 3 Kafka clusters per region (3 × N regions = 9-15 clusters globally) didn't pay for the marginal isolation — a single cluster with per-topic ACLs, per-topic compaction, per-topic retention is the cheaper-and-equally-isolated choice. Considered AWS Kinesis. Rejected because per-shot encoding fan-out (250 K work items / episode) needs >10 K partitions for parallel consumer scaling — Kinesis has hard shard limits that bind us into rebalance hell.

When it fails. Tentpole release lands all 8 episodes simultaneously; encoder queue depth blows to 2 M jobs; consumer lag goes from seconds to hours; bottom rungs ship on time but AV1-4K rungs ship 6 h late. Detection: kafka_consumer_lag_records, oldest_unconsumed_msg_age_seconds. Mitigation: priority partitions (tentpoles get reserved partitions that don't share with long-tail re-encodes); spot autoscale on the encoder consumer group; DLQ for poison chunks routed to encoder-dlq so one un-encodable chunk doesn't pin a worker into retry-loop. SECOND failure mode: outbox CDC relay lag — the asset-publish saga writes a row to metadata-db AND publishes a chunk-ready / cache-invalidation event via Kafka; if the CDC relay stalls (poison row, partition leader churn, schema-registry wedge), packager publishes never reach the catalogue cache invalidation consumer, and freshly-packaged titles stay invisible at the edge until the relay drains. Detection: oldest_unpublished_outbox_age, debezium_slot_lag_bytes. Mitigation: bound the WAL retention so the slot can't grow unbounded; alert at age > 30 s; switch to a fresh slot + replay from WAL archive if catastrophic.

Worker · Encoder FarmFFmpeg + x264/x265/libvpx-VP9/libaom-AV1/SVT-AV1; YouTube Argos VCU ASIC

Pulls encode-jobs from the transcode bus, downloads the source chunk from object-hot via MezzFS-style streaming mount (no full-download), runs the codec at the requested rung × codec, writes the encoded chunk back to object-hot, then publishes a chunk-ready event so the packager can stitch. Software encoders run on spot instances; the AV1 hot path runs on YouTube-class Argos VCU ASICs (10x more compute-efficient than CPU encoders).

Why it exists. Considered making the upload service kick off encodes synchronously. Rejected because (1) per-shot fan-out for a 30-min episode is ~250 K work items — the upload SLA can't wait, (2) encoding is CPU-bound and bursty (tentpoles) so the right tool is async with elastic capacity, (3) encoder failure modes (poison chunks, codec crashes) need a DLQ pattern that's natural in async but ugly in sync. Considered ASIC-only. Rejected because not all codec combinations are supported on Argos hardware yet (libaom-AV1 quality tuning still needs CPU); we run the hybrid.

When it fails. Tentpole queue backlog (Squid Game S2 lands all 8 episodes at midnight) — bottom rungs ship on time, top rungs ship 6 h late. Detection: encoder_queue_oldest_msg_age, sla_breach_count. Mitigation: priority queues (tentpoles get reserved encoder pool), DLQ for poison chunks. Codec regression: a bad libaom-AV1 build corrupts a fresh batch of titles before the SSIM/PSNR audit catches it. Detection: encode_psnr_p99, encode_ssim_p99. Mitigation: ringed encoder rollouts (1% / 10% / 50%) gated on quality SLOs.

Worker · Packager + DRMBento4 / Shaka Packager + CENC + Widevine/FairPlay/PlayReady

Consumes chunk-ready events from the transcode bus, fetches the encoded chunks for an asset from object-hot, stitches them into CMAF segments + master/media manifests, encrypts with CENC (CTR for Widevine/PlayReady, CBCS for FairPlay) using a content key, writes the packaged bundle back to object-hot, and writes the asset's published=true row to metadata-db (with the variants fingerprint + DRM key URLs). Once metadata-db acks, the title is live: catalogue cache invalidation propagates, edge fills can pick it up.

Why it exists. Considered doing packaging inside the encoder. Rejected because (1) packagers stitch ACROSS shot boundaries (encoder is per-shot), so a packager has to wait for all shots' chunks before finalizing a manifest, (2) DRM custody is sensitive — the content key wrapping and license-server handshake belong in a separate, hardened service, (3) packager bugs (the chunked-encode fence-post bug Netflix wrote about) produce A/V drift, and isolating that failure mode from encoder churn matters operationally. Considered shipping the packager output to the gateway directly. Rejected because publish must be transactional with metadata-db (asset visible iff packaged).

When it fails. Chunked-encode fence-post bug: chunk boundaries misaligned, stitching produces A/V drift > 40 ms — viewers report 'dialog out of sync'. Detection: pre-publish A/V sync probe (SSIM + audio waveform alignment), post-publish error reports. Mitigation: overlap-and-validate chunk boundaries at the encoder; A/V sync probe gates publish. DRM key rotation desync between encoder + license server: encoder ships ciphertext with key K1, license server still serving K0 → decryption fails on every player. Mitigation: atomic key bundles, ringed deploys, license caching on-device with a 24 h TTL so a brief desync doesn't affect existing sessions.

Worker · Popularity / Tier MoverApache Flink stream processing + tier-move scheduler

Consumes playback-events from the transcode bus (the player's session-start + bitrate-switch telemetry, ingested via the gateway). Computes per-title popularity scores per region with a sliding decay window (1 hr / 24 hr / 7 day / 30 day buckets). Writes scores to metadata-db. Drives two control loops: (1) every 6 h, demote bottom-decile titles from object-hot → cold-archive (storage lifecycle action). (2) when a back-catalogue title's 24 h score crosses a promote threshold, prefetch from cold-archive → object-hot so the next OCA fill window picks it up.

Why it exists. Considered keeping popularity scoring inside the catalogue cache directly. Rejected because the input stream (playback events at ~20 M events/sec) doesn't fit a cache's write pattern, and the windowed-aggregation logic needs a stream processor. Considered Spark batch (run nightly). Rejected because tier-promote latency matters for trending content — by the time a TikTok meme-trend hits 'titanic-rerelease' levels, you want the title hot within hours, not the next morning. Flink streaming gives us 5-minute granularity at acceptable cost.

When it fails. Glacier restore rate-limit: a TikTok-trending sports replay needs 500 historical titles restored simultaneously; restore queue grows to days. Detection: glacier_restore_queue_age, popularity_promote_lag. Mitigation: pre-emptive restore 24 h ahead via predictive trend signals; budget per-region per-day so a single trend can't pin the restore queue. Promote storm: every region promotes the same 50 titles within an hour after a global event — encoder farm gets re-encode burst. Mitigation: stagger promotes by region; reuse already-encoded variants whenever possible (no re-encode needed if we already have the right rungs).

Queue · Encoder DLQKafka tombstone topic + S3-archived sample chunks

Receives encode jobs that have failed > N retries (default 5) — corrupt mezzanine, codec crash, missing audio track, malformed timecode. Each entry persists the failing chunk's S3 key, codec parameters, error stack trace, and the originating titleId so a human can triage. Drains slowly via a manual ops review queue; auto-resurrects entries when the encoder ships a fix that resolves a class of failures (e.g. libaom-AV1 1.4 fixes a crash bug → all DLQ entries with that error replay).

Why it exists. Considered letting the consumer simply skip + log a poison chunk. Rejected because (1) silently dropping a chunk produces an A/V-misaligned final stream that ships to viewers, (2) Kafka's at-least-once semantics + idempotent encoder means retry is the expected path; with no DLQ, a single corrupt chunk pins one consumer in a retry-loop forever and hides capacity from healthy work. Considered using a separate Kafka cluster for DLQ. Rejected — same cluster, separate topic, separate consumer group is enough.

When it fails. DLQ fills past 100 K entries: indicates a systemic encoder regression (bad codec build, mezzanine validator bug). Detection: dlq_depth_total, dlq_growth_rate. Mitigation: pause the encoder fleet, roll back to last-known-good build, replay DLQ once the regression is confirmed fixed. Without the DLQ, the same poison chunks would silently retry forever and consume encoder capacity that healthy titles need.

MonitoringNetflix Atlas (in-memory dimensional TSDB) + Mantis stream processing + Conviva / NPAW for client-side QoE

Two streams of signals land here: (a) service-side metrics scraped from every named SLO in the canonical (manifest_p99, license_5xx_rate, gateway_active_streams, kafka_consumer_lag_records, glacier_restore_queue_age, encode_psnr_p99, oldest_unreplicated_lsn, dlq_depth_total) — Atlas serves the dashboards + the alert rules; (b) client-side QoE via Conviva / NPAW: average_delivered_bitrate, rebuffer_ratio, rung_switch_rate, concurrent_session_derivative. SPS (Started Plays per Second) is the load-bearing top-line.

Why it exists. Considered relying on cloud-provider monitoring (CloudWatch / Stackdriver). Rejected because (1) the dimensional cardinality of streaming KPIs (per-region × per-codec × per-rung × per-DRM × per-device-class) blows up cloud-monitoring quota costs at YouTube scale; (2) the JOIN-derivative + per-key QPS alerts need a streaming compute layer (Mantis), which CloudWatch doesn't provide; (3) the client-side QoE signal can't be derived from server-side metrics alone — Conviva / NPAW listen on the player's CDN-fetch results, not the origin's. Without this node, every failureMode claim in the canonical has nowhere to live.

When it fails. Monitor down → on-call goes blind for the duration. Detection: synthetic health check from a pager-bot every 30 s. Mitigation: warm DR Atlas in a second region, alert routing failover via PagerDuty's redundant ingestion endpoints. Gray failure: a metric pipeline regression silently drops dimensions — the dashboard renders but the alert never fires. Mitigation: synthetic 'inject + assert-fired' daily test on 5 critical alert rules.