Chapter 06 — Caching (The #1 Speed Trick)
Chapter 06 — Caching (The #1 Speed Trick)
Hey everyone! Welcome back to Namaste System Design! 🙏
If system design had a "most valuable player," it would be caching. It's the single most common way to make a slow system fast. There's even a famous joke: "There are only two hard things in computer science — cache invalidation and naming things." By the end of this chapter you'll understand both the magic and the pain.
What we will cover:
- What a cache is (the kitchen-counter analogy)
- Why RAM caching is 100x faster than disk
- Where caches live (browser → CDN → server → DB)
- Caching strategies: cache-aside, read-through, write-through, write-back, write-around
- Eviction policies: LRU, LFU, FIFO, TTL
- The hard part: invalidation, stale data & cache stampede
- Interview Questions
1. The Kitchen Counter Analogy
You're cooking. The pantry (database) is in the basement — going there takes 2 minutes. The counter (cache) is right next to you — grabbing something takes 2 seconds.
┌─────────────────────────────────────────────────────────────┐ │ Smart cook: keep the things you use OFTEN on the counter. │ │ │ │ Salt, oil, spices → COUNTER (cache, instant) │ │ Rare items → PANTRY (database, slow trip) │ │ │ │ You don't run to the basement for salt every single time. │ │ That's caching: keep hot data close and fast. │ └─────────────────────────────────────────────────────────────┘
Definition: a cache is a small, fast store (usually in RAM, like Redis or Memcached) that holds copies of frequently-used data so you avoid slow trips to the database.
2. Why It's Such a Big Deal
Remember the latency ladder from Chapter 03:
Read from RAM (cache) ~100 nanoseconds ⚡ ~100,000x faster Read from disk (DB) ~10 milliseconds 🐌 Turning a 50ms DB read into a 1ms cache read means: → users feel instant response → the database handles WAY fewer queries (it survives)
WITHOUT cache: WITH cache:
every request → DB 1st request → DB, store it
DB at 100% 🔥 next 10,000 → cache ⚡
DB barely touched 😌
3. Where Do Caches Live? (Everywhere!)
A single request can hit MANY caches on its way:
[ Browser cache ] → your own device remembers images/CSS
│ miss
[ CDN cache ] → a server near you holds static files (Ch 12)
│ miss
[ Load balancer / API cache ]
│ miss
[ Application cache ] → Redis/Memcached in front of the DB ← the classic
│ miss
[ Database ] → the slow source of truth (last resort)
Each layer that "hits" saves the trip to everything below it.
When people say "add a cache" in an interview, they usually mean the application cache (Redis) sitting between the app servers and the database.
4. Caching Strategies
How does data get INTO the cache, and how do writes work? These are the five patterns interviewers expect you to name and compare.
4.1 Cache-Aside (Lazy Loading)
The application talks to the cache itself. On a miss, the app — not the cache — goes and fetches from the DB and fills the cache.
1. App asks cache: "got user 1?"
HIT → return it ⚡ (done)
MISS → ↓
2. App reads DB, gets user 1
3. App stores user 1 in cache
4. Returns it. Next time → HIT.
✅ PROS Simple. Cache only holds data that's actually used.
❌ CONS First read for any item is slow (a "cold" miss).
If the app forgets step 3, the cache silently
never fills.
👉 USE IT WHEN You want full control and don't mind writing
the miss-handling code yourself. The default
pattern most teams reach for.
4.2 Read-Through
Same idea as cache-aside, but the cache library itself handles the miss — not your app code. The app only ever talks to the cache; the cache loads from the DB behind the scenes.
App ──"got user 1?"──▶ [ Cache ]
│ MISS
▼
Cache loads from DB itself
│
Cache stores it, returns to App
The app code never touches the DB directly.
✅ PROS Cleaner app code — the cache handles loading logic.
❌ CONS Needs a cache library/provider that supports it
(e.g. Redis behind a read-through wrapper).
👉 USE IT WHEN You want cache-aside's behaviour without
repeating miss-handling code everywhere.
4.3 Write-Through
Every write goes to the cache and the DB, at the same time, before the write is considered done.
App writes "price = ₹200"
│
├──▶ Cache updated ✅
└──▶ DB updated ✅
Only THEN does the app get "write successful".
Cache and DB are never out of sync.
✅ PROS Cache is always fresh. Reads right after a write
are correct.
❌ CONS Every write pays the cost of two writes → slightly
slower writes.
👉 USE IT WHEN Data must be correct immediately after a
write — e.g. account balances, inventory counts.
4.4 Write-Back (Write-Behind)
The write lands in the cache first. The DB is updated later, in a batch, in the background.
App writes "price = ₹200"
│
▼
Cache updated ✅ → app told "done" IMMEDIATELY
│
│ (later, batched)
▼
DB updated eventually
Writes feel instant. But right now, DB still says ₹100.
✅ PROS Fastest possible writes. Great for write-heavy
workloads (view counts, activity logs).
❌ CONS If the cache dies before flushing, those writes
are GONE. Data loss risk. 💥
👉 USE IT WHEN Losing the last few seconds of data is
acceptable — analytics counters, not bank balances.
4.5 Write-Around
The write goes straight to the DB, skipping the cache entirely. The cache only fills later, on a normal read (like cache-aside).
App writes "price = ₹200" ──▶ DB only. Cache untouched. Next READ of that item → cache MISS → loads fresh value from DB → cache now correct.
✅ PROS Cache isn't filled with data that's written once
and rarely read again (saves RAM).
❌ CONS First read after a write is always a miss — slower.
👉 USE IT WHEN Writes are frequent but that data isn't
re-read right away — e.g. bulk imports, logs.
Visual Comparison
| Strategy | Who handles misses? | Write speed | Fresh after write? | Risk |
|---|---|---|---|---|
| Cache-Aside | App | N/A (writes go to DB) | ✅ | Cold misses |
| Read-Through | Cache | N/A (writes go to DB) | ✅ | Cold misses |
| Write-Through | Cache | ❌ Slower | ✅ | None (safe) |
| Write-Back | Cache | ✅ Fastest | ✅ | ❌ Data loss on crash |
| Write-Around | Cache (on next read) | ✅ Fast | ❌ First read misses | Stale until re-read |
What Do Big Companies Actually Use?
Real systems mix strategies rather than picking one for everything.
Redis / Memcached The workhorses behind cache-aside and
read-through setups at almost every
big tech company.
Facebook (TAO) A cache-aside style graph cache sitting
in front of MySQL, serving billions of
reads per second.
CDNs (Cloudflare,
Akamai, CloudFront) Read-through style at the edge — miss
once near you, then every nearby user
gets a hit.
Default answer for an interview: "I'd start with cache-aside using Redis — it's simple and only caches what's actually requested. I'd switch to write-through if freshness right after a write really matters, and I'd only reach for write-back if I can tolerate losing the last few seconds of data."
5. Eviction Policies: The Cache is Small, So What Do We Kick Out?
RAM is limited. When the cache is full and new data arrives, something must go. Here are the four policies interviewers expect you to know.
5.1 LRU — Least Recently Used
Kick out whichever item hasn't been touched in the longest time.
Cache holds 3 slots. Access order so far: A, B, C
[A] [B] [C] ← all 3 slots full
Access B again → [A] [C] [B] (B moves to "freshest")
New item D arrives → A is least recently used → evict A
→ [C] [B] [D]
✅ PROS Matches real usage well — recently used data tends
to be used again soon.
❌ CONS A one-off scan (e.g. a big report) can evict lots
of genuinely hot data.
👉 USE IT WHEN You don't know access patterns in advance.
The default choice almost everywhere (Redis'
default eviction family). ⭐
5.2 LFU — Least Frequently Used
Kick out whichever item has been accessed the fewest times, not the longest ago.
Access counts right now:
A → 50 hits ⭐ very popular
B → 2 hits
C → 12 hits
New item D arrives → B has the fewest hits → evict B
✅ PROS Keeps genuinely popular items even if they weren't
JUST accessed (unlike LRU).
❌ CONS Extra bookkeeping (a counter per key). Old items
that were once popular can overstay their welcome.
👉 USE IT WHEN Popularity is stable over time — e.g. caching
the most-viewed products on an e-commerce site.
5.3 FIFO — First In, First Out
Kick out whichever item was added first — no matter how often it's used.
Added in order: A (1st), B (2nd), C (3rd)
New item D arrives → A was added first → evict A
→ regardless of whether A is still hot!
✅ PROS Dead simple to implement — just a queue.
❌ CONS Can evict very popular data just because it's old.
Rarely optimal on its own.
👉 USE IT WHEN Simplicity matters more than hit rate, or as
a building block inside a smarter policy.
5.4 TTL — Time To Live
Every item gets an expiry timer. When the timer runs out, it's evicted automatically — regardless of usage.
SET session:42 value=... TTL=60s t=0s stored, valid t=45s still valid, a read doesn't reset the clock t=60s key auto-deleted. Next read → MISS → reload.
✅ PROS Guarantees data can't go stale forever. Great
alongside another policy (LRU + TTL together).
❌ CONS Not about memory pressure — a TTL key can still be
evicted EARLY by LRU/LFU if space runs out first.
👉 USE IT WHEN Freshness has a natural deadline — session
tokens, OTPs, stock prices, weather data.
Visual Comparison
| Policy | Evicts based on | Extra bookkeeping | Best for |
|---|---|---|---|
| LRU | Recency | Access-order list | General purpose (default) |
| LFU | Frequency | Per-key counter | Stable popularity |
| FIFO | Insertion order | Simple queue | Simplicity |
| TTL | Time elapsed | Expiry timestamp | Data with a natural deadline |
6. The Hard Part: Stale Data & Invalidation
Here's the catch. The cache holds a copy. If the real data in the DB changes, the cache is now wrong (stale) until it's updated or expires.
DB: price = ₹200 (someone just changed it from ₹100) Cache: price = ₹100 ← STILL showing the old price! 😱 Users see ₹100. This is a STALE cache.
How we fight staleness:
✔ TTL (expiry) → cache auto-dies after N seconds, re-fetches
✔ Write-through → update cache whenever DB updates
✔ Explicit invalidation → on a write, delete the cache key so next
read re-loads fresh data
The eternal trade-off: longer TTL = faster & cheaper but more stale; shorter TTL = fresher but more DB load. How stale can this data be? A stock price: milliseconds. A user's profile photo: minutes are fine. Match the TTL to the need.
Watch out: cache stampede (thundering herd)
Popular key expires → 10,000 requests all MISS at once → all 10,000 hammer the DB simultaneously → DB melts. 🔥 Fix: "lock" so only ONE request rebuilds the cache while others wait, or stagger expiry times.
Interview Questions — Quick Fire!
Q: What is caching and why is it used?
"Caching stores copies of frequently accessed data in a fast layer, usually RAM like Redis, so we avoid repeatedly hitting a slower source like a database. It dramatically reduces latency — a RAM read is around a hundred thousand times faster than disk — and it reduces load on the database, letting the system serve far more traffic."
Q: Explain the cache-aside strategy.
"In cache-aside, the application first checks the cache. On a hit, it returns the cached data. On a miss, it reads from the database, stores the result in the cache, and returns it. So the cache is populated lazily — only with data that's actually requested. It's the most common pattern, and the trade-off is that the first read for any item is slow."
Q: What's the difference between read-through and cache-aside?
"They behave almost identically, but who owns the miss-handling code is different. In cache-aside, the application code checks the cache and, on a miss, goes to the database itself. In read-through, the cache library does that loading internally, so the application only ever talks to the cache. Read-through gives cleaner application code but needs a cache that supports it."
Q: When would you use write-back instead of write-through?
"Write-through writes to the cache and database together, so it's safe but slower. Write-back writes to the cache only and flushes to the database later in a batch, so writes are much faster but recent writes can be lost if the cache crashes before flushing. I'd use write-back for something like view counts or activity logs, where losing a few seconds of data is fine, and write-through for anything like balances where it isn't."
Q: What is cache invalidation and why is it hard?
"Cache invalidation is keeping the cache consistent with the source of truth when the underlying data changes. It's hard because the cache holds a copy — if the database updates but the cache doesn't, users see stale data. We handle it with TTL expiry, write-through updates, or explicitly deleting the cache key on writes, but choosing the right approach and timing is genuinely tricky."
Q: What is an eviction policy? Compare LRU and LFU.
"Since a cache has limited memory, an eviction policy decides what to remove when it's full. LRU — Least Recently Used — evicts the item that hasn't been accessed for the longest time, which works well when recent access predicts future access. LFU — Least Frequently Used — evicts the item with the fewest total accesses, which works better when popularity is stable over time rather than recency-based. LRU is simpler and more common; LFU needs a counter per key."
Q: What's a cache stampede and how do you prevent it?
"A cache stampede happens when a popular key expires and many simultaneous requests all miss and hit the database at once, potentially overwhelming it. We prevent it by locking so only one request rebuilds the cache while others wait, or by staggering expiry times so keys don't all expire together."
Key Points to Remember
| Concept | Key Takeaway |
|---|---|
| What | Fast RAM store (Redis) holding hot data to avoid slow DB trips. |
| Why | ~100,000x faster than disk; slashes latency AND database load. |
| Strategies | Cache-aside / read-through (default) · write-through (fresh) · write-back (fast, risky) · write-around (skip cache on write). |
| Eviction | LRU is the go-to (recency) · LFU (frequency) · FIFO (simplicity) · TTL (natural deadline). |
| The pain | Stale data. Fight it with TTL / write-through / invalidation. Watch for stampedes. |
What's Next?
Chapter 07 tackles database replication — keeping multiple copies of your data so reads are fast and your system survives a database crash. We'll meet the leader-follower pattern and the dreaded "replication lag."
Keep designing, keep scaling! See you in the next one!
Post a Comment