Deep Dive 03 — How Kafka Works Inside

Deep Dive 03 — How Kafka Works Inside

Hey everyone! Welcome to the final Deep Dive! 🙏

In Chapter 10 we met message queues and pub/sub, and kept mentioning a mysterious king: Kafka. It powers the event backbones of LinkedIn, Uber, Netflix — moving trillions of messages a day. But Kafka isn't quite a normal queue; it's a distributed commit log, and once that clicks, its power makes total sense. Let's open the hood.

What we will cover:

  • Kafka is a LOG, not a traditional queue
  • Topics, partitions, and offsets
  • Producers, consumers, and consumer groups
  • How partitions give massive parallelism
  • Replication & durability
  • Why Kafka is so fast
  • Interview Questions

1. The Big Idea: Kafka is an Append-Only Log

A normal queue (like RabbitMQ) deletes a message once it's consumed. Kafka is different: it's an append-only log — messages are written in order and kept (for days or forever). Consumers just track how far they've read.

┌─────────────────────────────────────────────────────────────┐
│   THINK OF A DIARY 📖 you only ADD to, never erase.         │
│                                                             │
│   Producers APPEND entries to the end.                      │
│   Consumers READ from wherever they left off (a bookmark).  │
│   The diary keeps everything — many readers, their own      │
│   bookmarks, at their own pace.                             │
└─────────────────────────────────────────────────────────────┘
   A LOG (ordered, append-only, each entry has an OFFSET):

   offset:  0    1    2    3    4    5    6  ← next append here
          [msg][msg][msg][msg][msg][msg]
                          ▲              ▲
                   consumer A's    consumer B's
                   bookmark(3)     bookmark(5)

   Both read the SAME log independently. Nothing is deleted
   when read. A can even "rewind" to offset 0 and re-read! 🔁

2. Topics, Partitions & Offsets

   TOPIC     = a named stream of messages (e.g. "orders").
   PARTITION = a topic is SPLIT into partitions — each is an
               independent ordered log. This is the key to scale.
   OFFSET    = a message's position number within a partition.

   Topic "orders" split into 3 partitions:
   ┌───────────────────────────────────────────────┐
   │ Partition 0:  [o0][o1][o2][o3]...              │
   │ Partition 1:  [o0][o1][o2]...                  │
   │ Partition 2:  [o0][o1][o2][o3][o4]...          │
   └───────────────────────────────────────────────┘

   Ordering is guaranteed WITHIN a partition, not across them.
   A message's partition is chosen by a key (e.g. hash(user_id))
   → all of one user's events land in the same partition, IN ORDER.

3. Producers, Consumers & Consumer Groups

   PRODUCER → appends messages to a topic (picks partition by key).

   CONSUMER GROUP → a team of consumers sharing the work of a topic.
   Kafka assigns each partition to ONE consumer in the group:

   Topic "orders" (3 partitions), consumer group with 3 workers:
        Partition 0 ──▶ Consumer A
        Partition 1 ──▶ Consumer B
        Partition 2 ──▶ Consumer C
   → 3× parallelism! Each worker owns its partitions.

   RULE: max useful consumers in a group = number of partitions.
   (4th consumer with only 3 partitions → sits idle.)
   → So you pick partition count based on desired parallelism.
   PUB/SUB for free: TWO different consumer groups both read the
   SAME topic independently (each with its own offsets):

        ┌─▶ Group "email-service"     (sends emails)
   topic─┤
        └─▶ Group "analytics-service" (counts stats)

   Both see EVERY message. That's broadcast + work-sharing in one.

4. Replication & Durability

   Each partition is REPLICATED across brokers (Kafka servers):
     • one LEADER replica (handles reads/writes)
     • several FOLLOWER replicas (copies for safety)

   Partition 0:  Leader on Broker 1
                 Followers on Broker 2, Broker 3

   Broker 1 dies? A follower is promoted to leader → no data lost,
   no downtime. (Same leader-follower idea as Chapter 07!)

   Messages are written to DISK (and replicated) before ack →
   durable. Kafka can safely retain terabytes for days/weeks.

5. Why Is Kafka So Fast?

   ✅ SEQUENTIAL disk writes → appending to a log is nearly as
      fast as RAM; no random seeks. (Disks are fast in a LINE.)
   ✅ ZERO-COPY → sends data from disk to network without
      copying through the app — huge efficiency.
   ✅ BATCHING → groups messages together to cut per-message overhead.
   ✅ PARTITIONS → horizontal parallelism across many brokers.
   ✅ CONSUMERS PULL → they read at their own pace; Kafka doesn't
      track per-message delivery state (the offset is the state).

   Result: millions of messages/sec on modest hardware. 🚀

Common uses

   • Event streaming / activity tracking (clicks, views)
   • Decoupling microservices (Ch 15) via events
   • Log aggregation, metrics pipelines
   • Feeding real-time analytics & data lakes
   • The async backbone behind fan-out, notifications, etc.

Interview Questions — Quick Fire!

Q: How is Kafka different from a traditional message queue?

"A traditional queue deletes a message once it's consumed. Kafka is an append-only distributed log — messages are written in order and retained for a configured period, and each consumer tracks its own offset, or position, in the log. This means multiple independent consumers can read the same data at their own pace, and a consumer can even rewind and reprocess old messages, which a normal queue can't do."

Q: What are topics, partitions, and offsets?

"A topic is a named stream of messages. Each topic is split into partitions, which are independent ordered logs — this is what lets Kafka scale horizontally and process in parallel. An offset is a message's sequential position within a partition. Ordering is guaranteed within a partition but not across partitions, and the partition for a message is usually chosen by hashing a key so related messages stay ordered together."

Q: How do consumer groups provide parallelism?

"A consumer group is a set of consumers sharing the work of a topic. Kafka assigns each partition to exactly one consumer in the group, so with three partitions and three consumers you get three-way parallelism. The maximum useful parallelism equals the number of partitions — extra consumers beyond that sit idle. Different consumer groups can each read the whole topic independently, which gives pub/sub broadcast alongside work-sharing."

Q: How does Kafka ensure durability?

"Each partition is replicated across multiple brokers with one leader and several followers. Messages are written to disk and replicated before being acknowledged, so if a broker fails, a follower is promoted to leader with no data loss and minimal downtime. This leader-follower replication, combined with disk persistence, lets Kafka reliably retain large volumes of data."

Q: Why is Kafka so fast?

"Several reasons: it uses sequential disk writes by appending to a log, which avoids slow random seeks; zero-copy transfers send data straight from disk to the network; it batches messages to reduce overhead; partitions give horizontal parallelism across brokers; and consumers pull at their own pace while Kafka just tracks offsets rather than per-message state. Together these let it handle millions of messages per second."


Key Points to Remember

ConceptKey Takeaway
Core ideaKafka is an append-only distributed log, not a delete-on-read queue.
Topic/partition/offsetTopics split into partitions (parallel ordered logs); offset = read position.
Consumer groupsOne partition per consumer → parallelism = partition count. Groups = pub/sub.
DurabilityPartitions replicated (leader-follower); disk-persisted before ack.
SpeedSequential writes + zero-copy + batching + partitions + pull consumers.

What's Next?

You've now opened all three black boxes! Head to the Bonus — Top System Design Interview Questions for a rapid-fire review of the whole series, then go ace that interview.

Keep designing, keep scaling! You've got this! 🚀