Chapter 22 — Design Video Streaming (YouTube / Netflix)
Chapter 22 — Design Video Streaming (YouTube / Netflix)
Hey everyone! Welcome to the grand finale of Season 5! 🙏
This is the boss level. Video is the heaviest workload on the internet — petabytes of storage, billions of viewers, and the unforgiving demand of zero buffering on everything from a fiber connection to a shaky 3G phone in a moving train. It brings together nearly every concept in this series. Let's design YouTube.
Step 1 — Clarify Requirements
FUNCTIONAL: ✔ Upload a video ✔ Watch a video (stream smoothly) ✔ Search / recommendations (out of scope today) NON-FUNCTIONAL: ✔ MASSIVE storage (petabytes) & bandwidth ✔ Low startup latency + NO buffering while watching ✔ Works on any device & any network speed ✔ Highly available globally ✔ Read-heavy: views ≫ uploads (by a huge margin) SCOPE: "Focus on the upload pipeline and the smooth watch path."
Step 2 — The Two Halves of the Problem
┌─────────────────────────────────────────────────────────────┐ │ VIDEO SYSTEMS = two very different problems: │ │ │ │ 1. UPLOAD & PROCESS (write path, slow, heavy, async) │ │ Store the raw file, then TRANSCODE it into many │ │ formats/qualities. │ │ │ │ 2. WATCH & STREAM (read path, must be FAST & smooth) │ │ Deliver the right quality from a server near the │ │ viewer, adapting to their network. │ └─────────────────────────────────────────────────────────────┘
Step 3 — The Upload & Transcoding Pipeline
When you upload a video, we can't just store one file. Phones, TVs, and slow networks all need different versions. So we transcode.
[ Creator ] ─upload raw 4K file─▶ [ Upload service ]
│ store raw in
▼ blob storage (S3)
[ Object Storage ]
│
put a job on a QUEUE (Ch 10)
▼
[ Transcoding workers ] (many!)
│ produce MANY versions:
▼
┌───────────┬───────────┬───────────┬───────────┐
4K/2160p 1080p 720p 480p 240p
(+ split each into small CHUNKS, e.g. 4-sec segments)
│
▼ push all versions to
[ CDN ] (Ch 12)
Transcoding is SLOW & CPU-heavy → done ASYNC via a queue by a fleet of workers. The uploader isn't blocked: they get "processing..." and a notification when it's ready. (Exactly the async pattern from Chapter 10.)
Step 4 — The Watch Path: CDN + Adaptive Streaming
Two ideas make streaming smooth for a billion people:
(a) CDN — bring the video close
Video is HUGE. Streaming it from one origin across oceans = buffering hell. So every video's chunks are cached on CDN edge servers WORLDWIDE (Chapter 12). A viewer in Tokyo streams from a Tokyo edge. This is ~90% of the magic. ⚡
(b) Adaptive Bitrate Streaming (ABR) — the anti-buffer trick
The video is pre-chopped into small CHUNKS (e.g. 4 seconds),
each available in EVERY quality. The player picks the quality
PER CHUNK based on your CURRENT network speed:
Fast wifi → [1080p][1080p][1080p] ...
Network dips→ [1080p][ 480p][ 240p] ... (drops quality,
keeps PLAYING)
Recovers → [ 480p][720p][1080p] ... (climbs back up)
→ Video keeps playing smoothly by trading QUALITY for
NO BUFFERING. This is HLS / MPEG-DASH. It's why YouTube
rarely freezes — it just gets blurrier for a second.
WATCH FLOW: 1. Click video → get a "manifest" listing chunks + qualities 2. Player requests chunk 1 at a quality matching your bandwidth 3. Chunks stream from the NEAREST CDN edge 4. Player continuously re-measures speed → adapts next chunk
Step 5 — High-Level Design (All Together)
UPLOAD SIDE WATCH SIDE
─────────── ──────────
[Creator] [Viewer]
│ upload │ watch
▼ ▼
[Upload svc] → [Object Storage] [App/API] → returns manifest
│ (raw + outputs) │ + CDN URLs
▼ queue ▼
[Transcode workers] [ CDN edges ] ⚡ stream chunks
│ → many qualities+chunks ▲
└────────► push to CDN ──────────┘
[ Metadata DB ] ← title, uploader, views, likes, manifest info
[ Message Queue ] ← decouples upload from transcoding
Step 6 — Bottlenecks & Trade-offs
⚠ Storage (petabytes) → cheap object storage (S3-like), not a DB.
Store video as chunks; metadata in a DB.
⚠ Bandwidth cost → CDN offloads origin; huge % served at edge.
⚠ Transcoding load → async worker fleet; scale workers with queue depth.
⚠ Popular vs long-tail→ hot videos cached on many edges; rarely-watched
ones pulled on demand (pull CDN, Ch 12).
⚠ Startup latency → pre-warm popular content on CDNs; small first chunk.
⚠ Metadata reads → cache + read replicas (views/likes are read-heavy).
⚠ Live streaming → similar but low-latency chunks + real-time transcode
(a harder variant worth mentioning).
Notice how this one design reuses almost the whole series: object storage, queues (Ch 10), CDN (Ch 12), caching (Ch 06), read replicas (Ch 07), async processing, and the read-heavy scaling mindset. That's the beauty of the finale.
Interview Questions — Quick Fire!
Q: Why do we transcode uploaded videos?
"Because viewers watch on wildly different devices and network speeds — a 4K TV, a phone on 3G, a laptop on wifi. We can't serve one giant file to all of them. Transcoding converts the uploaded video into multiple resolutions and bitrates, and splits each into small chunks, so the player can pick the right version for each viewer's device and current bandwidth."
Q: Why is transcoding done asynchronously?
"Transcoding is slow and CPU-intensive — a long video into many formats can take a while. Blocking the uploader on that would be a terrible experience. So we store the raw file, place a job on a message queue, and a fleet of worker servers processes it in the background. The uploader gets an immediate 'processing' response and a notification when the video is ready."
Q: How do you stream video smoothly without buffering?
"Two things. First, a CDN caches video chunks on edge servers worldwide, so viewers stream from a nearby server instead of a distant origin. Second, adaptive bitrate streaming: the video is pre-chunked into small segments available in every quality, and the player picks the quality for each chunk based on the viewer's current network speed. If the connection dips, it drops to a lower quality to keep playing rather than freezing, then climbs back up when the network recovers."
Q: Where do you store the actual video files?
"In cheap, durable object storage like Amazon S3, not a database — we're dealing with petabytes. The raw upload and all the transcoded chunk outputs live in object storage, distributed to CDN edges for delivery. The database only holds metadata like title, uploader, view counts, and the manifest that lists the available chunks and qualities."
Q: How do you handle popular versus rarely-watched videos?
"Popular videos are proactively cached across many CDN edges so they're always close to viewers. Long-tail videos that are rarely watched don't need to sit on every edge — they're pulled from the origin on demand and cached temporarily. This balances performance for hot content against the cost of caching everything everywhere."
Key Points to Remember
| Concept | Key Takeaway |
|---|---|
| Two halves | Upload+transcode (async, heavy) vs watch+stream (fast, smooth). |
| Transcoding | Convert to many qualities + chunks, async via queue + worker fleet. |
| CDN | Cache chunks worldwide; stream from nearest edge — ~90% of the magic. |
| Adaptive bitrate | Per-chunk quality by live bandwidth → trades quality for no buffering (HLS/DASH). |
| Storage | Video in cheap object storage; metadata in a DB. Petabyte scale. |
What's Next? — You Did It! 🎉
That's the full journey — from "what is a server?" to designing YouTube. You now have the vocabulary, the trade-off instinct, and a repeatable framework. The Deep Dives are waiting when you want to open the hardest black boxes: consistent hashing, back-of-the-envelope estimation, and how Kafka works inside. And don't miss the Bonus interview-questions round.
Go draw some boxes. You're ready. Keep designing, keep scaling! 🚀
Post a Comment