Chapter 03 — Latency, Throughput & Availability
Chapter 03 — Latency, Throughput & Availability
Hey everyone! Welcome back to Namaste System Design! 🙏
Interviewers love throwing numbers at you: "How do you get p99 latency under 100ms?" or "We need four nines of availability." If those words make you nod nervously without understanding — this chapter fixes that forever. These are the vital signs of any system, like blood pressure and heart rate for a body.
What we will cover:
- Latency — how fast is one request?
- Throughput — how many requests at once?
- Why they're different (and often fight)
- Percentiles: p50, p95, p99 — and why averages lie
- Availability and the famous "nines"
- The latency numbers every engineer should know
- Interview Questions
1. Latency vs Throughput — The Highway Analogy
Picture a highway. 🛣️
┌─────────────────────────────────────────────────────────────┐ │ LATENCY = how long ONE car takes to cross the highway │ │ (measured in time: ms, seconds) │ │ → "SPEED" for a single request │ │ │ │ THROUGHPUT = how many cars cross PER HOUR │ │ (measured in rate: requests/sec) │ │ → "CAPACITY" of the whole system │ └─────────────────────────────────────────────────────────────┘
Here's the twist that confuses everyone: they are independent. A highway can have low latency (empty road, cars zoom) but low throughput (only 1 lane). Or high latency (traffic jam) but high throughput (16 lanes moving slowly but carrying tons of cars).
Add more LANES (servers) → throughput UP, latency same Raise the SPEED LIMIT (faster code/cache) → latency DOWN Too many cars (overload) → latency UP (jam), throughput plateaus
2. Latency: The Whole Journey
When we say a request took 200ms, that time is spent in many places:
USER CLICK ──▶ [ network ] ──▶ [ load balancer ] ──▶ [ server ]
30ms 1ms code: 10ms
│
▼
[ database ]
50ms
◀────────────── total round trip ≈ 200ms ──────────────
To cut latency, find the BIGGEST slice and attack it.
(Usually: the database or the network.)
This is why caching (Chapter 06) is magic — it removes the 50ms database slice for repeated reads.
3. Averages Lie — Meet Percentiles
Suppose 100 users load a page. 99 take 50ms, but 1 takes 5000ms (5 sec). The average is ~100ms — sounds fine! But one real user just had a terrible 5-second experience. Averages hide your worst pain.
┌─────────────────────────────────────────────────────────────┐ │ PERCENTILES — read the real experience │ ├─────────────────────────────────────────────────────────────┤ │ │ │ p50 (median) = 50% of requests are faster than this │ │ → "typical" user │ │ │ │ p95 = 95% are faster; the slowest 5% are worse │ │ │ │ p99 = 99% are faster; only the worst 1% are slower │ │ → this is where angry users live 😤 │ │ │ └─────────────────────────────────────────────────────────────┘
Golden rule: optimize for the tail (p95/p99), not the average. At scale, that "worst 1%" is millions of real people. This is called tail latency, and taming it is a mark of senior design thinking.
4. Availability — The Famous "Nines"
Availability = the % of time your system is up and working. We measure it in "nines."
| Availability | Nickname | Downtime per YEAR | Downtime per DAY |
|---|---|---|---|
| 99% | "two nines" | ~3.65 days | ~14 minutes |
| 99.9% | "three nines" | ~8.75 hours | ~86 seconds |
| 99.99% | "four nines" | ~52 minutes | ~8.6 seconds |
| 99.999% | "five nines" | ~5.25 minutes | ~0.86 seconds |
See how brutal it gets? Going from three nines to five nines means shrinking yearly downtime from ~9 hours to ~5 minutes. Each extra nine costs dramatically more money (redundancy, failovers, on-call teams). So we ask: "How many nines does this feature actually deserve?" A payment system needs five; a blog's "related posts" widget needs two.
HOW WE BUY AVAILABILITY: ──────────────────────── ✔ Redundancy → multiple servers, so one dying isn't fatal ✔ No single → remove every single-point-of-failure point of failure ✔ Replication → copies of data in multiple places ✔ Failover → auto-switch to a backup when primary dies ✔ Health checks → detect a dead node fast and route around it
Availability vs Reliability — Don't Confuse Them
AVAILABILITY = "Is it up right now?" (uptime %)
RELIABILITY = "Does it behave correctly (no data loss,
and consistently?" no wrong answers)
A system can be AVAILABLE but unreliable (up, but giving
wrong data) — that's often worse than being down!
5. Latency Numbers Every Engineer Should Know
You don't memorize these to the nanosecond — you memorize the orders of magnitude, so you know what's cheap vs expensive.
Read from CPU cache (L1) ~1 nanosecond Read from RAM (memory) ~100 nanoseconds ← FAST Read from SSD ~100 microseconds Read from a fast network (same ~500 microseconds datacenter round trip) Read from spinning hard disk ~10 milliseconds ← SLOW Network round trip across the ~150 milliseconds world (e.g. India ↔ USA) ← SLOWEST BIG LESSON: Memory ≫ SSD ≫ Disk ≫ Cross-continent network. This is WHY we cache in RAM and WHY we use CDNs (Chapter 12).
Interview Questions — Quick Fire!
Q: What's the difference between latency and throughput?
"Latency is the time for a single request to complete — a measure of speed, in milliseconds. Throughput is how many requests the system handles per unit of time — a measure of capacity, in requests per second. They're independent: you can improve throughput by adding servers without changing latency, or reduce latency with caching without changing throughput."
Q: Why do we look at p99 latency instead of the average?
"Because averages hide the worst experiences. If 99% of requests are fast but 1% take five seconds, the average still looks good, yet at scale that 1% is millions of frustrated users. The p99 tells you what your slowest users actually experience, so optimizing the tail latency gives a truer picture of quality."
Q: What does 99.99% availability mean in practical terms?
"Four nines means the system is down at most about 52 minutes per year, or roughly 8.6 seconds per day. Each additional nine cuts allowed downtime by about 10x and costs significantly more in redundancy and operations, so we match the number of nines to how critical the feature is."
Q: How do you increase a system's availability?
"By eliminating single points of failure through redundancy — running multiple servers and database replicas — and adding automatic failover so a backup takes over when a primary dies. Health checks detect dead nodes quickly so traffic is routed around them. The goal is that no single component's failure can bring the whole system down."
Key Points to Remember
| Concept | Key Takeaway |
|---|---|
| Latency | Time for ONE request (speed). Attack the biggest slice — often DB or network. |
| Throughput | Requests per second (capacity). More servers/lanes → more throughput. |
| Percentiles | Optimize p95/p99 (the tail), not the misleading average. |
| Availability | Uptime %. Nines cost exponentially; match to how critical the feature is. |
| Availability ≠ Reliability | Up vs. correct. A system can be up yet giving wrong answers. |
| Latency ladder | RAM ≫ SSD ≫ Disk ≫ cross-world network. Cache in RAM; use CDNs. |
What's Next?
In Chapter 04 we meet the traffic cop that made horizontal scaling possible: the Load Balancer. How does it split traffic? What algorithms does it use? What happens when it becomes the bottleneck itself? Let's find out.
Keep designing, keep scaling! See you in the next one!
Post a Comment