Chapter 03 — Latency, Throughput & Availability
Chapter 03 — Latency, Throughput & Availability
Hey everyone! Welcome back to Namaste System Design! 🙏
Interviewers love throwing numbers at you: "How do you get p99 latency under 100ms?" or "We need four nines of availability." If those words make you nod nervously without understanding — this chapter fixes that forever. These are the vital signs of any system, like blood pressure and heart rate for a body.
What we will cover:
- Latency — how fast is one request?
- Throughput — how many requests at once?
- Why they're different (and often fight)
- Percentiles: p50, p95, p99 — and why averages lie
- Availability and the famous "nines"
- The latency numbers every engineer should know
- Interview Questions
1. Latency vs Throughput — The Highway Analogy
Picture a highway. 🛣️
┌─────────────────────────────────────────────────────────────┐ │ LATENCY = how long ONE car takes to cross the highway │ │ (measured in time: ms, seconds) │ │ → "SPEED" for a single request │ │ │ │ THROUGHPUT = how many cars cross PER HOUR │ │ (measured in rate: requests/sec) │ │ → "CAPACITY" of the whole system │ └─────────────────────────────────────────────────────────────┘
Here's the twist that confuses everyone: they are independent. A highway can have low latency (empty road, cars zoom) but low throughput (only 1 lane). Or high latency (traffic jam) but high throughput (16 lanes moving slowly but carrying tons of cars).
Add more LANES (servers) → throughput UP, latency same Raise the SPEED LIMIT (faster code/cache) → latency DOWN Too many cars (overload) → latency UP (jam), throughput plateaus
2. Latency: The Whole Journey
When we say a request took 200ms, that time is spent in many places:
USER CLICK ──▶ [ network ] ──▶ [ load balancer ] ──▶ [ server ]
30ms 1ms code: 10ms
│
▼
[ database ]
50ms
◀────────────── total round trip ≈ 200ms ──────────────
To cut latency, find the BIGGEST slice and attack it.
(Usually: the database or the network.)
This is why caching (Chapter 06) is magic — it removes the 50ms database slice for repeated reads.
3. Averages Lie — Meet Percentiles
Suppose 100 users load a page. 99 take 50ms, but 1 takes 5000ms (5 sec). The average is ~100ms — sounds fine! But one real user just had a terrible 5-second experience. Averages hide your worst pain. To see the real experience, sort every request from fastest to slowest and pick specific positions — that's a percentile.
3.1 p50 (Median) — The Typical User
Sort 100 requests fastest to slowest. The value sitting at position 50 is the p50.
position: 1 ... 50 ... 95 99 100
time (ms): 20 ... 45 ... 220 1800 5000
▲
p50 = 45ms
Half of all requests were faster than 45ms. Half were slower.
👉 Tells you: what a "normal" user actually experiences.
3.2 p95 — Where the Tail Begins
The value at position 95. 95% of requests were faster than this; the slowest 5% were worse.
position: 1 ... 50 ... 95 99 100
time (ms): 20 ... 45 ... 220 1800 5000
▲
p95 = 220ms
1 in every 20 requests was slower than 220ms.
👉 Use it when: you want to catch a growing problem before it hits "most" of your users.
3.3 p99 — Where Angry Users Live 😤
The value at position 99. Only the worst 1% of requests are slower than this.
position: 1 ... 50 ... 95 99 100
time (ms): 20 ... 45 ... 220 1800 5000
▲
p99 = 1800ms
At 1,000,000 requests/day, that "worst 1%" is
10,000 real people having a bad time — every single day.
👉 Golden rule: optimize for the tail (p95/p99), not the average. This is called tail latency, and taming it is a mark of senior design thinking.
4. Availability — The Famous "Nines"
Availability = the % of time your system is up and working. We measure it in "nines." Let's go through each level one by one — these come up by name in almost every interview.
4.1 Two Nines — 99%
The system is allowed to be down 1% of the time.
99% uptime = ~3.65 days of downtime PER YEAR = ~14 minutes of downtime PER DAY
👉 Use it when: an internal tool, a hobby project, a blog's "related posts" widget. Nobody's day is ruined if it blips.
4.2 Three Nines — 99.9%
99.9% uptime = ~8.75 hours of downtime PER YEAR = ~86 seconds of downtime PER DAY
👉 Use it when: most everyday consumer apps. Good enough that users rarely notice a blip.
4.3 Four Nines — 99.99%
99.99% uptime = ~52 minutes of downtime PER YEAR = ~8.6 seconds of downtime PER DAY
👉 Use it when: a serious production service — e-commerce checkout, a messaging app. This is where most "big" systems aim.
4.4 Five Nines — 99.999%
99.999% uptime = ~5.25 minutes of downtime PER YEAR = ~0.86 seconds of downtime PER DAY
👉 Use it when: banking, payments, telecom carriers, emergency systems. Getting here costs serious money — multi-region failover, 24/7 on-call, redundant everything.
Visual Comparison
| Nines | Availability | Downtime / Year | Downtime / Day | Typical use |
|---|---|---|---|---|
| Two nines | 99% | ~3.65 days | ~14 min | ❌ real product, ✅ internal tools |
| Three nines | 99.9% | ~8.75 hrs | ~86 sec | ✅ everyday consumer apps |
| Four nines | 99.99% | ~52 min | ~8.6 sec | ✅ serious production services |
| Five nines | 99.999% | ~5.25 min | ~0.86 sec | ✅ banking, payments, telecom |
See how brutal it gets? Going from three nines to five nines means shrinking yearly downtime from ~9 hours to ~5 minutes. Each extra nine costs dramatically more money (redundancy, failovers, on-call teams). So we ask: "How many nines does this feature actually deserve?" A payment system needs five; a blog's "related posts" widget needs two.
HOW WE BUY AVAILABILITY: ──────────────────────── ✔ Redundancy → multiple servers, so one dying isn't fatal ✔ No single → remove every single-point-of-failure point of failure ✔ Replication → copies of data in multiple places ✔ Failover → auto-switch to a backup when primary dies ✔ Health checks → detect a dead node fast and route around it
Availability vs Reliability — Don't Confuse Them
AVAILABILITY = "Is it up right now?" (uptime %)
RELIABILITY = "Does it behave correctly (no data loss,
and consistently?" no wrong answers)
A system can be AVAILABLE but unreliable (up, but giving
wrong data) — that's often worse than being down!
5. Latency Numbers Every Engineer Should Know
You don't memorize these to the nanosecond — you memorize the orders of magnitude, so you know what's cheap vs expensive.
Read from CPU cache (L1) ~1 nanosecond Read from RAM (memory) ~100 nanoseconds ← FAST Read from SSD ~100 microseconds Read from a fast network (same ~500 microseconds datacenter round trip) Read from spinning hard disk ~10 milliseconds ← SLOW Network round trip across the ~150 milliseconds world (e.g. India ↔ USA) ← SLOWEST BIG LESSON: Memory ≫ SSD ≫ Disk ≫ Cross-continent network. This is WHY we cache in RAM and WHY we use CDNs (Chapter 12).
Interview Questions — Quick Fire!
Q: What's the difference between latency and throughput?
"Latency is the time for a single request to complete — a measure of speed, in milliseconds. Throughput is how many requests the system handles per unit of time — a measure of capacity, in requests per second. They're independent: you can improve throughput by adding servers without changing latency, or reduce latency with caching without changing throughput."
Q: Why do we look at p99 latency instead of the average?
"Because averages hide the worst experiences. If 99% of requests are fast but 1% take five seconds, the average still looks good, yet at scale that 1% is millions of frustrated users. The p99 tells you what your slowest users actually experience, so optimizing the tail latency gives a truer picture of quality."
Q: How would you actually compute p95 from raw data?
"Sort every recorded request time from fastest to slowest, then pick the value sitting at the 95th percentile position — for 100 samples that's roughly the 95th value in the sorted list. That value tells you the threshold below which 95% of requests fell. In practice you'd use a metrics tool like Prometheus or Datadog to compute this continuously rather than sorting by hand."
Q: What does 99.99% availability mean in practical terms?
"Four nines means the system is down at most about 52 minutes per year, or roughly 8.6 seconds per day. Each additional nine cuts allowed downtime by about 10x and costs significantly more in redundancy and operations, so we match the number of nines to how critical the feature is."
Q: How do you increase a system's availability?
"By eliminating single points of failure through redundancy — running multiple servers and database replicas — and adding automatic failover so a backup takes over when a primary dies. Health checks detect dead nodes quickly so traffic is routed around them. The goal is that no single component's failure can bring the whole system down."
Key Points to Remember
| Concept | Key Takeaway |
|---|---|
| Latency | Time for ONE request (speed). Attack the biggest slice — often DB or network. |
| Throughput | Requests per second (capacity). More servers/lanes → more throughput. |
| Percentiles | Sort requests, pick a position. p50 = typical user, p95/p99 = the tail. Optimize the tail, not the average. |
| Availability | Two nines (~14 min/day down) → five nines (~0.86 sec/day down). Cost rises exponentially per nine; match to how critical the feature is. |
| Availability ≠ Reliability | Up vs. correct. A system can be up yet giving wrong answers. |
| Latency ladder | RAM ≫ SSD ≫ Disk ≫ cross-world network. Cache in RAM; use CDNs. |
What's Next?
In Chapter 04 we meet the traffic cop that made horizontal scaling possible: the Load Balancer. How does it split traffic? What algorithms does it use? What happens when it becomes the bottleneck itself? Let's find out.
Keep designing, keep scaling! See you in the next one!
Post a Comment