Chapter 07 — Database Replication
Chapter 07 — Database Replication
Hey everyone! Welcome back to Namaste System Design! 🙏
Your app servers are horizontally scaled, you've added a cache — but everything still funnels into one database. That single DB is now your biggest weakness: if it's slow, everyone's slow; if it dies, everyone's down. The first fix is replication — keeping copies. Let's see how.
What we will cover:
- What replication is (the photocopy analogy)
- Replication types: single-leader, multi-leader, leaderless
- How it scales reads and boosts availability
- Replication lag & the "read-your-own-write" problem
- Synchronous vs asynchronous replication
- Interview Questions
1. The Photocopy Analogy
Imagine one precious library book everyone wants to read. With one copy, there's a huge queue, and if it's lost, the knowledge is gone forever. So the library makes photocopies. 📚
┌─────────────────────────────────────────────────────────────┐ │ REPLICATION = keeping multiple copies of your database. │ │ │ │ Many copies → many people read at once (no queue) │ │ → one copy burns? others survive (safety) │ └─────────────────────────────────────────────────────────────┘
2. Replication Types
Who is allowed to accept writes? That single question splits replication into three types. Let's go through each — with a concrete failure story for each.
2.1 Single-Leader (Leader-Follower)
One leader takes all writes. Several followers copy the leader and serve reads. This is the classic, default setup — also called master-slave or primary-replica.
WRITES
[ App ] ──────────────────▶ [ LEADER DB ] (the only writer)
│
replicates changes to ↓ ↓ ↓
┌──────────────────┬──────────────────┐
[ Follower 1 ] [ Follower 2 ] [ Follower 3 ]
▲ ▲ ▲
└─────── READS ────┴──────────────────┘
[ App ] reads from any follower
Writes → ONE leader. Reads → MANY followers.
FAILURE STORY: the leader dies mid-afternoon.
Writes have nowhere to go until a follower is PROMOTED
to be the new leader ("failover"). If it's automatic,
this takes seconds. If manual, an on-call engineer gets
paged at 3am. Either way, writes are blocked until then.
👉 USE IT WHEN You want a simple, well-understood default.
Most relational databases (Postgres, MySQL)
start here.
2.2 Multi-Leader
More than one node can accept writes — often one leader per data center or region. Each leader replicates its writes to the others.
[ Leader: Mumbai ] ◀──sync──▶ [ Leader: London ] User in India writes to Mumbai. User in UK writes to London. Both regions get fast LOCAL writes.
FAILURE STORY: the same row, edited in two places at once.
User A (Mumbai) sets stock quantity = 5.
User B (London) sets stock quantity = 8.
...at the SAME time, before either sync reaches the other.
Which one wins?? This is a WRITE CONFLICT. Someone has to
resolve it — "last write wins" (simple but can silently
drop a real update), or custom merge logic (complex but safer).
👉 USE IT WHEN You need fast local writes in multiple regions
and can tolerate — or resolve — conflicts.
2.3 Leaderless
No leader at all. Any node can accept a write. The client writes to several nodes directly and reads from several nodes, then reconciles.
[ App ] writes to Node A, Node B, Node C (all at once) [ App ] reads from Node A, Node B (majority wins) Quorum rule: R + W > N N = total nodes, W = nodes that must ack a write, R = nodes that must respond to a read. e.g. N=3, W=2, R=2 → 2+2 > 3 → always overlaps by 1 node → a read is guaranteed to see at least one up-to-date copy.
FAILURE STORY: a node was down during a write, then comes back.
Node C missed the last write (it was offline).
A later read hits Node C and Node A → they DISAGREE.
The system uses version numbers/timestamps to figure out
which value is newer ("read repair") and fixes Node C.
👉 USE IT WHEN You need very high availability and can
accept eventual consistency. Cassandra and
DynamoDB both work this way.
Visual Comparison
| Type | Who writes? | Conflicts? | Availability | Examples |
|---|---|---|---|---|
| Single-Leader | One leader only | ❌ None | Blocked until failover | Postgres, MySQL (default) |
| Multi-Leader | Several leaders | ✅ Possible | High (local writes) | Multi-region SQL setups |
| Leaderless | Any node | ✅ Possible (resolved via quorum) | Highest | Cassandra, DynamoDB |
For most interviews, single-leader is the default you describe; bring up multi-leader and leaderless to show depth.
3. Why This Is Powerful
✅ READ SCALING → most apps are read-heavy (e.g. 100 reads
per 1 write). Add followers → handle more
reads without touching the leader.
✅ AVAILABILITY → leader dies? PROMOTE a follower to become
the new leader (this is "failover").
✅ GEO-LOCALITY → put a follower near your users (India, US,
EU) so reads are fast everywhere.
✅ BACKUPS/ANALYTICS → run heavy reports on a follower so the
main DB isn't slowed down.
4. The Catch: Replication Lag
Copying from leader to followers isn't instant. For a few milliseconds (sometimes seconds under load), a follower is behind the leader. This gap is replication lag, and it causes a famous bug:
THE "READ-YOUR-OWN-WRITE" PROBLEM
─────────────────────────────────
1. You post a comment → written to LEADER ✅
2. Page reloads, reads → from a FOLLOWER that hasn't
received the copy yet
3. Your own comment is MISSING! 😱 "Did it fail??"
...then a second later it appears. Confusing and bad UX.
Fixes:
✔ Read-your-writes: send a user's reads to the LEADER for a few seconds right after they write. ✔ Route critical reads (e.g. your own profile) to the leader. ✔ "Monotonic reads": keep a user pinned to one follower so they never see data go backwards.
5. Synchronous vs Asynchronous Replication
Once you know WHO can write, the next question is: does the leader wait for followers to confirm before saying "done"?
5.1 Synchronous Replication
The leader waits for one or more followers to confirm they've stored the write, before telling the app "success."
App writes X
│
▼
Leader stores X ──▶ Follower stores X ──▶ Follower ACKs
│ │
└──────────── only THEN: "success" ◀───────┘
Slower — the app waits for the round trip.
FAILURE STORY: the leader dies AFTER the follower confirmed.
Because the follower already has the data, the promoted
follower has the SAME value the app was told was saved.
No data lost. This is the whole point of paying the
latency cost.
👉 USE IT WHEN Losing even one write is unacceptable —
financial ledgers, orders, critical records.
5.2 Asynchronous Replication
The leader replies "done" immediately, then copies to followers afterward, whenever it gets to it.
App writes X
│
▼
Leader stores X ──▶ "success" (app moves on immediately)
│
│ (happens after, in the background)
▼
Follower eventually stores X
FAILURE STORY: the leader dies BEFORE replicating.
The app was told "success". The leader crashes one second
later, before the write reached any follower. A follower
gets promoted — and it never had that write. The write the
user was told succeeded is now GONE. 😱
👉 USE IT WHEN Speed matters more than the (small) risk of
losing the last write or two. What most systems
use, because leader crashes are rare.
A common middle ground: semi-synchronous — the leader waits for one follower to confirm (safety) but not all of them (speed).
Visual Comparison
| Synchronous | Semi-Sync | Asynchronous | |
|---|---|---|---|
| Waits for | All followers | One follower | No one |
| Data safety | ✅ No loss | ✅ Mostly safe | ❌ Can lose recent writes |
| Write speed | ❌ Slowest | Medium | ✅ Fastest |
| Used by | Zero-loss systems | Balanced setups | Most systems (speed wins) |
Interview Questions — Quick Fire!
Q: What is database replication and why do we use it?
"Replication keeps multiple copies of a database in sync across different machines. We use it for two main reasons: to scale reads, since most apps are read-heavy and we can spread reads across many replicas; and for high availability, because if one database fails, another copy can take over. It also enables geo-local reads and offloading analytics."
Q: Explain leader-follower replication.
"One node is the leader and handles all writes. It streams its changes to one or more followers, which are read-only copies. Reads are distributed across the followers to scale read capacity, while all writes go through the single leader to keep them consistent. If the leader dies, a follower is promoted to become the new leader."
Q: What's the difference between multi-leader and leaderless replication?
"In multi-leader replication, several nodes can each accept writes — often one leader per region — and they sync with each other, which risks write conflicts when the same record is edited in two places at once. In leaderless replication, any node accepts writes and reads, and consistency is maintained using quorums, where the number of nodes that must acknowledge a write plus the number that must respond to a read is greater than the total number of nodes. Leaderless systems like Cassandra and DynamoDB prioritize availability and use read repair to fix nodes that missed a write."
Q: What is replication lag and what problem does it cause?
"Replication lag is the delay between a write landing on the leader and appearing on the followers. It causes the read-your-own-writes problem: a user writes data, then reads from a follower that hasn't caught up, so their own change seems to be missing. We fix it by routing a user's reads to the leader briefly after they write, or pinning them to one replica."
Q: What's the difference between synchronous and asynchronous replication?
"With synchronous replication, the leader waits for followers to confirm a write before acknowledging it — this guarantees no data loss but makes writes slower. With asynchronous replication, the leader acknowledges immediately and copies to followers afterward — writes are fast, but recent writes can be lost if the leader crashes before replicating. Most systems use async, sometimes semi-sync as a compromise."
Key Points to Remember
| Concept | Key Takeaway |
|---|---|
| Replication | Multiple synced copies of the DB → read scaling + availability. |
| Replication types | Single-leader (simple, default) · multi-leader (fast local writes, conflicts) · leaderless (highest availability, quorums). |
| Replication lag | Followers trail the leader → read-your-own-writes bug. Route fresh reads to leader. |
| Sync vs async | Sync = safe but slow · Async = fast but can lose recent writes · Semi-sync = middle ground. |
| Limit | Replication scales READS, not writes or storage size — that needs sharding. |
What's Next?
Replication copies the whole database, so it scales reads but not writes or total size. When your data is simply too big for one machine, you must split it. Chapter 08 covers sharding & partitioning — the art of chopping one giant database into many.
Keep designing, keep scaling! See you in the next one!
Post a Comment