Chapter 14 — Consistency & Consensus
Chapter 14 — Consistency & Consensus
Hey everyone! Welcome to Season 4 — Reliability & Scale! 🙏
In Chapter 09 (CAP) we said systems sometimes trade consistency for availability. But "consistency" isn't one thing — it's a spectrum. And when machines must agree on a decision despite crashes and network chaos, they need a consensus algorithm. This is the deep, brainy heart of distributed systems. We'll keep it simple and visual, with one running story: you post a comment, and refresh the page.
What we will cover:
- The consistency spectrum: strong → eventual
- Strong, eventual, read-your-writes, monotonic reads, and causal consistency — each with the comment story
- The consensus problem (agreeing despite failure)
- Quorums: the R + W > N trick, with worked numbers
- Paxos vs Raft in plain English
- Interview Questions
1. Consistency Is a Spectrum, Not a Switch
STRONG ◀─────────────────────────────────────────▶ WEAK
consistency consistency
[Strong] every read sees the latest write, always.
Feels like ONE machine. Safe but slower.
→ bank balances, inventory
[Eventual] reads might be stale for a moment, but ALL
copies converge to the same value soon.
Fast & highly available.
→ likes, view counts, DNS, social feeds
┌─────────────────────────────────────────────────────────────┐ │ STRONG: write ₹500 → EVERY reader instantly sees ₹500 │ │ │ │ EVENTUAL: write ₹500 → some readers briefly see the old │ │ value → within ~1s, everyone sees ₹500 │ └─────────────────────────────────────────────────────────────┘
2. Consistency Models, One by One
Let's use one story throughout: you post a comment on a friend's photo, then refresh the page. Each model answers "what do you see?" differently.
2.1 Strong Consistency
Every read, from anyone, anywhere, sees the most recent write. No stale copies, ever.
You: post comment "Nice pic!" → write is CONFIRMED You: refresh → see it ✅ (of course) Friend (other continent): refreshes 1ms later → sees it too ✅ The system makes it look like there's only ONE copy of the data, even though it might be replicated across continents. That guarantee costs coordination time.
👉 Use it when: the cost of showing stale data is high — bank balances, inventory counts, seat bookings.
2.2 Eventual Consistency
Writes propagate in the background. Any read taken right now might be stale, but every replica converges to the same value eventually.
You: post comment "Nice pic!" → write lands on Server A
Friend: refreshes 200ms later, hits Server B → doesn't
see it yet (replication hasn't reached B)
Friend: refreshes again 1 second later → NOW sees it ✅
(B has caught up)
Nobody was harmed by that 1-second gap. That's eventual
consistency earning its keep — fast writes, high
availability, in exchange for a brief, harmless delay.
👉 Use it when: a moment of staleness is harmless — likes, view counts, comment counts, social feeds.
2.3 Read-Your-Writes (Read-After-Write) Consistency
A specific, narrower promise: you will always see your own writes immediately, even if the system is only eventually consistent for everyone else.
You: post comment "Nice pic!" → write lands on Server A
You: refresh IMMEDIATELY → the system routes YOUR read
back to Server A (or checks a cache tagged with your
write) → you see your own comment ✅, instantly
Friend: refreshes at the same moment, hits Server B →
might still see the OLD state for another moment.
You never have to wonder "did my post actually go through?"
— this fixes exactly the bug we hit back in Chapter 07.
👉 Use it when: users must trust that their own action was saved — posting, editing a profile, submitting a form.
2.4 Monotonic Reads
Once you've seen a value, you will never see an older one on a later read — time doesn't go backward for you, even if different replicas are at different points.
Friend: refreshes, hits Server B (already replicated)
→ SEES your comment ✅
Friend: refreshes again 2 seconds later, this time hits
Server C (a slow replica, still catching up)
→ WITHOUT monotonic reads: comment disappears! 😱
→ WITH monotonic reads: system remembers what this
friend already saw, and routes them to a replica
at least as fresh, or serves from a sticky source
→ comment stays visible ✅
The promise isn't "always fresh" — it's "never goes backward."
👉 Use it when: a value disappearing and reappearing would confuse or scare users — anything shown as a running state (comment counts, order status).
2.5 Causal Consistency
If one event caused another, everyone must see them in that order. Unrelated events can arrive in any order — only cause-and-effect pairs are protected.
Friend posts: "What time does the show start?" You reply: "8pm!" WITHOUT causal consistency: a third viewer's feed could show "8pm!" BEFORE the question it's answering. Confusing! 😕 WITH causal consistency: the system tracks that your reply CAUSALLY DEPENDS on the question, and guarantees the question is visible first, everywhere. Unrelated events (someone else's unrelated comment on a different post) have no ordering requirement at all.
👉 Use it when: ordering of cause-and-effect matters for the content to make sense — comment threads, chat replies — but you don't need the expense of full strong consistency for everything.
Visual Comparison
| Model | Can a read be stale? | Your own writes visible instantly? | Can a value "go backward"? | Cause before effect? |
|---|---|---|---|---|
| Strong | ❌ Never | ✅ | ❌ Never | ✅ |
| Eventual | ✅ Yes, briefly | ❌ Not guaranteed | ✅ Possible | ❌ Not guaranteed |
| Read-your-writes | ✅ For others | ✅ Guaranteed for you | ✅ Possible for others | ❌ Not guaranteed |
| Monotonic reads | ✅ Yes, briefly | ❌ Not guaranteed | ❌ Never, for you | ❌ Not guaranteed |
| Causal | ✅ Yes, briefly | ❌ Not guaranteed | ✅ Possible | ✅ Guaranteed |
What Do Real Databases Use?
Google Spanner Strong (external) consistency globally,
using synchronized atomic clocks (TrueTime).
DynamoDB, Cassandra Tunable — you pick consistency level per
query (e.g. quorum reads/writes, Chapter
14 §4), trading latency for freshness.
MongoDB Strong within a replica set's primary;
eventual on secondary/replica reads unless
you opt into stronger read concerns.
Postgres/MySQL Strong on the primary; read replicas are
read replicas eventually consistent (replication lag).
Interview-safe default: "I'd pick the weakest consistency model the data can tolerate. Money and inventory need strong consistency. Social counters and feeds are fine with eventual. And read-your-writes is almost always worth adding on top, because users trust the system more when their own actions show up instantly."
3. The Consensus Problem
Now the hard part. Multiple nodes must agree on a single value — like "who is the leader?" or "what's the next entry in the log?" — even though nodes can crash and messages can be lost or delayed.
┌─────────────────────────────────────────────────────────────┐ │ CONSENSUS = getting a group of unreliable nodes to agree │ │ on ONE value, reliably. │ │ │ │ Analogy: a group of friends deciding on a restaurant over │ │ a bad phone line where calls keep dropping — yet they must │ │ ALL end up at the SAME place. Harder than it sounds! │ └─────────────────────────────────────────────────────────────┘
Consensus algorithms (Paxos, Raft) power the most critical systems: leader election, distributed locks, config stores like ZooKeeper and etcd (which runs Kubernetes).
4. Quorums — Agreement by Majority
The simplest consensus idea: majority vote. If more than half agree, that's the decision. This gives us the famous quorum formula:
N = total replicas
W = nodes that must ACK a WRITE
R = nodes that must respond to a READ
THE MAGIC RULE: R + W > N
→ guarantees your read set and write set OVERLAP by at
least one node, so a read always sees the latest write.
Example: N=3, W=2, R=2 → 2 + 2 = 4 > 3 ✅
Write reaches 2 nodes; any read of 2 nodes hits at least
one that has the new value. Consistency without contacting
ALL nodes → survives one node being down.
Write to nodes: [A✅] [B✅] [C ] (W=2, C missed it)
Read from nodes: [A✅] [C ] (R=2)
↑ overlap! A has the latest value ✅
Bigger cluster, same rule. Say N=5: you could use W=3, R=3 (3+3=6 > 5, tolerates 2 nodes down), or go cheaper with W=4, R=2 (4+2=6 > 5, fast reads but writes need 4 of 5 to ack). The formula stays the same — you're just choosing where to pay the cost, on reads or on writes.
5. Consensus Algorithms: Paxos vs Raft
5.1 Paxos — the original, and famously confusing
Paxos was the first widely studied consensus algorithm. It's provably correct, but its rules and roles (proposers, acceptors, learners) are notoriously hard to explain and implement correctly — even experienced engineers get it wrong. It's respected, but rarely what a team builds from scratch today.
5.2 Raft — designed to be understandable
Raft was designed specifically to fix Paxos's biggest problem: nobody could understand it. The core idea:
1. LEADER ELECTION
Nodes elect ONE leader by majority vote. Each node waits
a random time, then says "vote for me!". First to get a
majority becomes leader. (Random timers avoid ties.)
2. LOG REPLICATION
All writes go to the LEADER. The leader appends to its
log and replicates to followers. Once a MAJORITY confirm,
the entry is "committed" (safe, permanent).
3. FAILURE
Leader crashes? Followers notice (no heartbeat) → hold a
new election → new leader. The majority rule ensures no
committed data is ever lost.
[ Leader ] ──append "x=5"──▶ [ Follower 1 ] ✅
│ └──▶ [ Follower 2 ] ✅
│ majority (2 of 3) confirmed → COMMITTED ✅
heartbeats keep followers from starting an election
You won't implement Raft in an interview, but knowing "leader election + majority-committed log" shows real depth.
Visual Comparison
| Paxos | Raft | |
|---|---|---|
| Understandability | ❌ Notoriously hard | ✅ Designed to be teachable |
| Leader concept | Implicit / per-round | ✅ Explicit single leader |
| Used in the wild | Chubby, some internal Google systems | ✅ etcd, Consul, CockroachDB, TiKV |
| Core guarantee | Majority agreement | Majority agreement |
Interview Questions — Quick Fire!
Q: What's the difference between strong and eventual consistency?
"With strong consistency, every read reflects the most recent write — the system behaves as if there's a single copy, which is safe but slower and less available. With eventual consistency, reads may briefly return stale data, but all replicas converge to the same value shortly after. Eventual consistency trades immediate accuracy for speed and availability, which is fine for things like like-counts or feeds."
Q: Is eventual consistency a bad thing?
"Not at all — it's a deliberate trade-off. For data where a brief delay is harmless, like social media likes or view counts, eventual consistency buys much better performance and availability. You only need strong consistency where staleness is dangerous, like account balances or inventory. Choosing the right level per data type is the skill."
Q: What is read-your-writes consistency, and why does it matter?
"It guarantees that a user always sees their own writes immediately, even in an otherwise eventually consistent system — you post a comment and it's there when you refresh, even if a stranger elsewhere might briefly see the old state. It matters because users lose trust in a product if their own actions seem to vanish, even if that staleness would be harmless for other people's data."
Q: What's the difference between eventual consistency and monotonic reads?
"Eventual consistency only promises replicas will converge eventually — a read can be stale, and there's no promise about which replica you'll hit next. Monotonic reads adds a narrower guarantee on top: once you've seen a value, you will never see an older one on a later read, even if you're routed to a different, less-caught-up replica. It prevents data from appearing to 'go backward' for a given user."
Q: What is causal consistency?
"It guarantees that if one event causally depends on another — like a reply depending on the question it answers — every reader sees them in that cause-then-effect order. Unrelated events have no ordering guarantee. It's a middle ground: cheaper than strong consistency, but avoids the confusing 'answer before the question' problem that plain eventual consistency allows."
Q: What is consensus in distributed systems?
"Consensus is getting a group of nodes to agree on a single value despite failures — crashed nodes, lost or delayed messages. It's needed for things like electing a leader, distributed locks, and keeping a replicated log in sync. Algorithms like Paxos and Raft solve it, and they underpin critical systems like ZooKeeper and etcd."
Q: Explain quorum and the R + W > N rule.
"In a system with N replicas, a write must be acknowledged by W nodes and a read must gather responses from R nodes. If R plus W is greater than N, the read set and write set are guaranteed to overlap on at least one node, so a read always sees the latest write. This gives consistency without contacting every node, and it tolerates some nodes being down. You can also tune where the cost falls — high W and low R for fast reads, or the reverse for fast writes."
Q: How does Raft differ from Paxos, and how does Raft work at a high level?
"Paxos is the original consensus algorithm and provably correct, but it's famously hard to understand and implement correctly. Raft was designed specifically to be understandable while providing the same guarantees. Raft elects a single leader by majority vote using randomized timers to avoid ties. All writes go through the leader, which appends them to its log and replicates to followers; an entry is committed once a majority confirm it. If the leader fails, followers detect the missing heartbeat and elect a new leader. The majority requirement ensures committed data is never lost — that's why Raft is what most new systems, like etcd and CockroachDB, actually implement."
Key Points to Remember
| Concept | Key Takeaway |
|---|---|
| Spectrum | Strong (always latest, slower) ↔ eventual (briefly stale, fast). Pick per data type. |
| Strong | Every reader sees the latest write instantly. For money, inventory, bookings. |
| Eventual | Not broken — a deliberate scale/availability trade. Everywhere in social systems. |
| Read-your-writes | You always see your own writes immediately, even if others might not yet. |
| Monotonic reads | A value never appears to go backward for a given reader. |
| Causal | Cause is always seen before effect; unrelated events are unordered. |
| Consensus | Unreliable nodes agreeing on one value despite crashes. Powers leader election, locks. |
| Quorum | R + W > N guarantees read/write overlap → consistency with fault tolerance. |
| Paxos vs Raft | Same guarantees; Raft is the understandable one most new systems actually use. |
What's Next?
Chapter 15 tackles the architectural debate of the decade: monolith vs microservices. One big app or many small ones? We'll cut through the hype and give you the real trade-offs.
Keep designing, keep scaling! See you in the next one!
Post a Comment