Chapter 04 — Load Balancers
Chapter 04 — Load Balancers
Hey everyone! Welcome back to Namaste System Design! 🙏
In Chapter 02 we drew a mysterious box labelled [ LB ] that magically split traffic across servers. Today we open that box. The load balancer is the traffic cop of the internet — without it, horizontal scaling simply doesn't work. It's on the whiteboard in every single system design interview.
What we will cover:
- What a load balancer does (the receptionist analogy)
- How it makes many servers look like one
- Balancing algorithms: round robin, least connections, hashing
- Health checks — routing around dead servers
- Layer 4 vs Layer 7 load balancing
- "But isn't the LB itself a single point of failure?"
- Interview Questions
1. The Receptionist Analogy
Walk into a big hospital. You don't wander the halls hunting for a free doctor. You go to one reception desk, and the receptionist sends you to whichever doctor is free. 🛎️
┌─────────────────────────────────────────────────────────────┐ │ The LOAD BALANCER is that receptionist. │ │ │ │ Patients (users) all arrive at ONE address. │ │ The receptionist (LB) decides which doctor (server) │ │ handles each one — so no doctor is overwhelmed and │ │ no patient waits in the wrong line. │ └─────────────────────────────────────────────────────────────┘
┌──▶ [ Server 1 ] (30% busy)
[ Users ] ─────▶ [ LB ]──▶ [ Server 2 ] (32% busy)
one public IP └──▶ [ Server 3 ] (28% busy)
Users see ONE address. The LB hides the fleet behind it.
Add/remove servers anytime — users never notice. ✨
2. What the Load Balancer Buys You
✅ SCALABILITY → spread load so no server melts ✅ AVAILABILITY → a server dies? LB stops sending it traffic ✅ FLEXIBILITY → add/remove servers with zero user impact ✅ ONE ENTRY POINT → users hit one address, not 50 IPs ✅ (bonus) SSL termination, compression, sometimes caching
3. How Does It Choose a Server? (Algorithms)
The receptionist needs a rule for picking the next doctor. That rule is called a load balancing algorithm. Let's go through them one by one — these are the ones interviewers actually ask about.
3.1 Round Robin — the default
Requests are handed out one by one, in order. Then it loops back to the start.
Servers: A B C req 1 → A req 2 → B req 3 → C req 4 → A ← back to the top req 5 → B req 6 → C
✅ PROS Dead simple. Perfectly equal distribution.
❌ CONS Ignores how busy a server actually is.
Bad if one server is weaker than the others.
👉 USE IT WHEN All servers have the same hardware.
3.2 Weighted Round Robin — bigger server, bigger share
Same as round robin, but each server gets a weight based on its capacity. More powerful server → more requests.
Server A = weight 3 (beefy machine) Server B = weight 2 Server C = weight 1 (small machine) Order of requests: A A A B B C → then repeat Out of every 6 requests: A gets 3, B gets 2, C gets 1.
👉 Use it when: your fleet is a mix of big and small machines.
3.3 Least Connections — send it to whoever is least busy
The LB looks at how many requests each server is currently handling, and picks the smallest number.
Active connections right now:
A → 120 😰
B → 30 😌 ← fewest
C → 55
New request → B
👉 GREAT FOR Long-running requests
WebSockets
Chat apps
Video streaming
(Round robin would keep piling requests onto A even
though A is drowning. Least connections won't.)
3.4 Weighted Least Connections — capacity + busyness
A combination of 3.2 and 3.3. The LB looks at both the server's weight and its current connections.
Server A: weight = 4, connections = 80 Server B: weight = 1, connections = 30 B has fewer connections... ...but A is 4x more powerful, so 80 connections is light work for it. The LB does the maths on both before routing.
👉 Use it when: servers have different sizes and requests have different durations.
3.5 Least Response Time — pick the fastest
The LB keeps measuring how quickly each server replies, and sends the next request to the quickest one.
Average response time:
A → 35 ms
B → 120 ms 🐌
C → 18 ms ⚡ ← fastest
Next request → C
👉 Use it when: server performance changes over time (noisy neighbours, GC pauses, uneven workloads).
3.6 IP Hash — same user, same server
The LB hashes the client's IP address, and that hash decides the server. Same IP → same server, every time.
Client IP: 192.168.1.20
│
hash( )
│
▼
[ Server B ]
That client will ALWAYS land on Server B
(as long as the server list doesn't change).
👉 GREAT FOR Sticky sessions
Shopping carts kept in server memory
Login sessions kept in server memory
3.7 Random — just pick one
Exactly what it sounds like. Pick a server at random.
req 1 → C req 2 → A req 3 → B
Rarely used on its own — but surprisingly decent at scale, and it's the base of the "power of two choices" trick (pick 2 servers at random, send to whichever has fewer connections).
3.8 Consistent Hashing — hashing that survives change
Plain IP hash has an ugly problem: add or remove one server and almost every client gets remapped. Consistent hashing fixes that by placing servers and clients on a logical ring.
PLAIN HASH: 3 servers → 4 servers
= nearly ALL clients move 💥 (cache wiped)
CONSISTENT HASHING: 3 servers → 4 servers
= only ~1/4 of clients move ✅
👉 Use it when: distributed caches (Redis clusters), CDNs, or any service where remapping is expensive. We'll build the ring step by step in Deep Dive 01.
Visual Comparison
| Algorithm | Checks load? | Knows server capacity? | Sticky? | Best for |
|---|---|---|---|---|
| Round Robin | ❌ | ❌ | ❌ | Equal servers |
| Weighted Round Robin | ❌ | ✅ | ❌ | Different server sizes |
| Least Connections | ✅ | ❌ | ❌ | Long-lived connections |
| Weighted Least Connections | ✅ | ✅ | ❌ | Mixed-capacity fleets |
| Least Response Time | ✅ | Indirectly | ❌ | Dynamic workloads |
| IP Hash | ❌ | ❌ | ✅ | User sessions |
| Random | ❌ | ❌ | ❌ | Simple setups |
| Consistent Hashing | Partial | ❌ | ✅ (stable mapping) | Caches, distributed systems |
What Do Big Companies Actually Use?
Here's the honest answer: almost nobody relies on a single algorithm. Real systems combine the algorithm with health checks, server capacity and live metrics.
NGINX Round Robin (default), Least Connections,
IP Hash, Least Time (commercial version)
HAProxy Round Robin, Least Connections, Source/IP Hash,
Random, URI Hash
AWS / GCP / Azure
Optimised internal algorithms — usually
"least outstanding requests" + latency +
health + capacity, not plain round robin.
Default answer for an interview: "Round robin is the simplest and works when servers are identical. I'd switch to least-connections when request durations vary widely — like WebSockets or streaming — and use consistent hashing when I need stable mapping, such as a distributed cache."
4. Health Checks — The Life Saver
How does the LB know a server died? It constantly pings each one.
LB ──"you alive?"──▶ [ Server 2 ] ──"yes! 200 OK"──▶ ✅ keep it LB ──"you alive?"──▶ [ Server 3 ] ──(no reply)────▶ ❌ pull it out Every few seconds. The moment S3 stops replying, the LB STOPS sending it users. When S3 recovers, it's added back. Users never see the dead server. THIS is how "self-healing" and high availability actually happen.
5. Layer 4 vs Layer 7
Load balancers work at two different "depths" of understanding the traffic.
| Layer 4 (Transport) | Layer 7 (Application) | |
|---|---|---|
| Sees | Just IPs & ports (TCP/UDP) | The full HTTP request (URL, headers, cookies) |
| Can it read the URL? | No | Yes |
| Speed | Faster (less to inspect) | Slightly slower (reads content) |
| Smart routing | No | Yes — /images → image servers, /api → API servers |
| Example | AWS NLB | AWS ALB, Nginx |
Simple takeaway: Layer 7 is smarter (it understands HTTP, so it can route by path or do SSL termination); Layer 4 is faster and dumber. Most web apps use Layer 7.
6. "Isn't the LB Itself a Single Point of Failure?"
Brilliant catch — and interviewers love asking it. If all traffic flows through one LB and that LB dies... total outage. So we make the LB redundant too:
┌─▶ [ LB 1 - active ] ─┐
[ Users ] ─DNS─ ├─▶ [ server fleet ]
└─▶ [ LB 2 - standby ] ─┘
Two (or more) load balancers. If the active one dies,
the standby takes over its IP (a "floating IP") in seconds.
Big cloud LBs (AWS ELB) do this for you automatically.
The lesson generalizes: anywhere you have exactly one of something critical, you have a single point of failure — so you duplicate it.
Interview Questions — Quick Fire!
Q: What is a load balancer and why do we need one?
"A load balancer sits in front of a group of servers and distributes incoming requests across them. We need it because horizontal scaling only works if traffic is spread evenly — it prevents any single server from being overwhelmed, provides one entry point that hides the server fleet, and improves availability by routing traffic away from failed servers."
Q: Name a few load balancing algorithms.
"Round robin cycles through servers in order and is simplest for equal servers. Weighted round robin gives more powerful servers more traffic. Least connections sends the next request to the server with the fewest active connections, which is better when request durations vary. IP hash maps a client consistently to the same server, useful for session stickiness."
Q: How does a load balancer know a server is down?
"Through health checks — it periodically sends a request, like a ping to a health endpoint, and expects a healthy response. If a server stops responding, the load balancer removes it from the pool and stops routing traffic to it, then adds it back once it recovers. This is what makes the system self-healing."
Q: What's the difference between Layer 4 and Layer 7 load balancing?
"Layer 4 operates at the transport level, routing based only on IP and port without reading the request content — it's fast but not content-aware. Layer 7 operates at the application level, so it can read the HTTP request and route based on URL path, headers, or cookies, and do things like SSL termination. Layer 7 is smarter; Layer 4 is faster."
Q: Isn't the load balancer itself a single point of failure?
"It can be, so we make it redundant — typically an active-standby or active-active pair. If the active load balancer fails, a standby takes over its IP address within seconds using a floating IP. Managed cloud load balancers handle this redundancy automatically."
Key Points to Remember
| Concept | Key Takeaway |
|---|---|
| Purpose | Spreads traffic across servers; one entry point hiding the fleet. |
| Algorithms | Round robin (simple) · weighted RR (mixed sizes) · least connections (uneven load) · least response time (dynamic) · IP hash (stickiness) · consistent hashing (caches). |
| Health checks | Pings servers; pulls dead ones out automatically → self-healing. |
| L4 vs L7 | L4 = fast, IP/port only · L7 = smart, understands HTTP paths/headers. |
| LB redundancy | Duplicate the LB (active-standby) so it isn't itself a single point of failure. |
What's Next?
Season 1 done — you now understand scale, the vital-sign numbers, and the traffic cop. In Chapter 05 we start Season 2: Data & Storage with the eternal question — SQL vs NoSQL — but this time through a designer's eyes: which one to pick, and why, for a given system.
Keep designing, keep scaling! See you in the next one!
Post a Comment