Deep Dive 03 — How Kafka Works Inside
Deep Dive 03 — How Kafka Works Inside
Hey everyone! Welcome to the final Deep Dive! 🙏
In Chapter 10 we met message queues and pub/sub, and kept mentioning a mysterious king: Kafka. It powers the event backbones of LinkedIn, Uber, Netflix — moving trillions of messages a day. But Kafka isn't quite a normal queue; it's a distributed commit log, and once that clicks, its power makes total sense. Let's open the hood.
What we will cover:
- Kafka is a LOG, not a traditional queue
- Topics, partitions, and offsets
- Producers, consumers, and consumer groups
- How partitions give massive parallelism
- Replication & durability
- Why Kafka is so fast
- Interview Questions
1. The Big Idea: Kafka is an Append-Only Log
A normal queue (like RabbitMQ) deletes a message once it's consumed. Kafka is different: it's an append-only log — messages are written in order and kept (for days or forever). Consumers just track how far they've read.
┌─────────────────────────────────────────────────────────────┐ │ THINK OF A DIARY 📖 you only ADD to, never erase. │ │ │ │ Producers APPEND entries to the end. │ │ Consumers READ from wherever they left off (a bookmark). │ │ The diary keeps everything — many readers, their own │ │ bookmarks, at their own pace. │ └─────────────────────────────────────────────────────────────┘
A LOG (ordered, append-only, each entry has an OFFSET):
offset: 0 1 2 3 4 5 6 ← next append here
[msg][msg][msg][msg][msg][msg]
▲ ▲
consumer A's consumer B's
bookmark(3) bookmark(5)
Both read the SAME log independently. Nothing is deleted
when read. A can even "rewind" to offset 0 and re-read! 🔁
2. Topics, Partitions & Offsets
TOPIC = a named stream of messages (e.g. "orders").
PARTITION = a topic is SPLIT into partitions — each is an
independent ordered log. This is the key to scale.
OFFSET = a message's position number within a partition.
Topic "orders" split into 3 partitions:
┌───────────────────────────────────────────────┐
│ Partition 0: [o0][o1][o2][o3]... │
│ Partition 1: [o0][o1][o2]... │
│ Partition 2: [o0][o1][o2][o3][o4]... │
└───────────────────────────────────────────────┘
Ordering is guaranteed WITHIN a partition, not across them.
A message's partition is chosen by a key (e.g. hash(user_id))
→ all of one user's events land in the same partition, IN ORDER.
3. Producers, Consumers & Consumer Groups
PRODUCER → appends messages to a topic (picks partition by key).
CONSUMER GROUP → a team of consumers sharing the work of a topic.
Kafka assigns each partition to ONE consumer in the group:
Topic "orders" (3 partitions), consumer group with 3 workers:
Partition 0 ──▶ Consumer A
Partition 1 ──▶ Consumer B
Partition 2 ──▶ Consumer C
→ 3× parallelism! Each worker owns its partitions.
RULE: max useful consumers in a group = number of partitions.
(4th consumer with only 3 partitions → sits idle.)
→ So you pick partition count based on desired parallelism.
PUB/SUB for free: TWO different consumer groups both read the
SAME topic independently (each with its own offsets):
┌─▶ Group "email-service" (sends emails)
topic─┤
└─▶ Group "analytics-service" (counts stats)
Both see EVERY message. That's broadcast + work-sharing in one.
4. Replication & Durability
Each partition is REPLICATED across brokers (Kafka servers):
• one LEADER replica (handles reads/writes)
• several FOLLOWER replicas (copies for safety)
Partition 0: Leader on Broker 1
Followers on Broker 2, Broker 3
Broker 1 dies? A follower is promoted to leader → no data lost,
no downtime. (Same leader-follower idea as Chapter 07!)
Messages are written to DISK (and replicated) before ack →
durable. Kafka can safely retain terabytes for days/weeks.
5. Why Is Kafka So Fast?
✅ SEQUENTIAL disk writes → appending to a log is nearly as
fast as RAM; no random seeks. (Disks are fast in a LINE.)
✅ ZERO-COPY → sends data from disk to network without
copying through the app — huge efficiency.
✅ BATCHING → groups messages together to cut per-message overhead.
✅ PARTITIONS → horizontal parallelism across many brokers.
✅ CONSUMERS PULL → they read at their own pace; Kafka doesn't
track per-message delivery state (the offset is the state).
Result: millions of messages/sec on modest hardware. 🚀
Common uses
• Event streaming / activity tracking (clicks, views) • Decoupling microservices (Ch 15) via events • Log aggregation, metrics pipelines • Feeding real-time analytics & data lakes • The async backbone behind fan-out, notifications, etc.
Interview Questions — Quick Fire!
Q: How is Kafka different from a traditional message queue?
"A traditional queue deletes a message once it's consumed. Kafka is an append-only distributed log — messages are written in order and retained for a configured period, and each consumer tracks its own offset, or position, in the log. This means multiple independent consumers can read the same data at their own pace, and a consumer can even rewind and reprocess old messages, which a normal queue can't do."
Q: What are topics, partitions, and offsets?
"A topic is a named stream of messages. Each topic is split into partitions, which are independent ordered logs — this is what lets Kafka scale horizontally and process in parallel. An offset is a message's sequential position within a partition. Ordering is guaranteed within a partition but not across partitions, and the partition for a message is usually chosen by hashing a key so related messages stay ordered together."
Q: How do consumer groups provide parallelism?
"A consumer group is a set of consumers sharing the work of a topic. Kafka assigns each partition to exactly one consumer in the group, so with three partitions and three consumers you get three-way parallelism. The maximum useful parallelism equals the number of partitions — extra consumers beyond that sit idle. Different consumer groups can each read the whole topic independently, which gives pub/sub broadcast alongside work-sharing."
Q: How does Kafka ensure durability?
"Each partition is replicated across multiple brokers with one leader and several followers. Messages are written to disk and replicated before being acknowledged, so if a broker fails, a follower is promoted to leader with no data loss and minimal downtime. This leader-follower replication, combined with disk persistence, lets Kafka reliably retain large volumes of data."
Q: Why is Kafka so fast?
"Several reasons: it uses sequential disk writes by appending to a log, which avoids slow random seeks; zero-copy transfers send data straight from disk to the network; it batches messages to reduce overhead; partitions give horizontal parallelism across brokers; and consumers pull at their own pace while Kafka just tracks offsets rather than per-message state. Together these let it handle millions of messages per second."
Key Points to Remember
| Concept | Key Takeaway |
|---|---|
| Core idea | Kafka is an append-only distributed log, not a delete-on-read queue. |
| Topic/partition/offset | Topics split into partitions (parallel ordered logs); offset = read position. |
| Consumer groups | One partition per consumer → parallelism = partition count. Groups = pub/sub. |
| Durability | Partitions replicated (leader-follower); disk-persisted before ack. |
| Speed | Sequential writes + zero-copy + batching + partitions + pull consumers. |
What's Next?
You've now opened all three black boxes! Head to the Bonus — Top System Design Interview Questions for a rapid-fire review of the whole series, then go ace that interview.
Keep designing, keep scaling! You've got this! 🚀
Post a Comment