Lesson 7 · Reliability core

Replication & ISR

How Kafka survives a broker dying — and what acks=all was really waiting for.

Your win: explain leaders, followers, and the in-sync replica set; describe what happens when a broker holding a leader dies; and connect it back to the acks knob from Lesson 4.

Every partition is copied N times

A partition isn't stored on one broker — it's stored on replication factor brokers. One replica is the leader (handles all reads and writes); the rest are followers that continuously fetch from the leader to stay identical.1 Producers and consumers only ever talk to the leader; followers are pure backups.

partition 0, replication factor = 3 broker 1: [LEADER] ←── producers write here, consumers read here broker 2: (follower) ── fetches from leader broker 3: (follower) ── fetches from leader └── caught-up followers + leader = the ISR

ISR: the replicas that are allowed to take over

The ISR (in-sync replicas) is the leader plus every follower currently caught up to it. Kafka tracks this set live: a follower that falls too far behind is dropped from the ISR, and rejoins when it catches up.1

The ISR matters for one reason: when a leader dies, the new leader is elected from the ISR — and only from the ISR, because only those replicas have all the committed data. Followers that were lagging aren't eligible (electing one would lose data). The group of consumers reading that partition then just reconnects to the new leader and continues.

Replication factor = fault tolerance budget With replication factor N, you can lose up to N−1 brokers and still have every partition available. That's the durability Kafka is famous for — it's built entirely on leader + follower + ISR.

This is what acks=all was waiting for

Now Lesson 4 closes. acks=all means "confirm the write only once the whole ISR has it." So even if the leader dies immediately after, a caught-up follower becomes leader with your write intact. acks=1 (our setting) confirms after only the leader — faster, but if the leader dies before a follower copied it, that write is gone.

The real durability combo acks=all alone isn't enough — pair it with min.insync.replicas ≥ 2. That forces at least two replicas to hold a write before it's acknowledged, so a single broker loss never loses data. "acks=all + min.insync.replicas=2" is a gold-star interview answer.

Why your local stack shows none of this

Locally you'd never see replication in action, and the repo tells you why — topic creation forces replication factor 1 for the local environment:

internal/golibs/kafka/kafka.go:773 (UpsertTopic) 
if isLocalEnv {
    topicConfig.ReplicationFactor = 1   // one copy — no fault tolerance locally
}
conn.CreateTopics(*topicConfig)
Anchor Real replication factors (and partition counts) for our topics are declared per service in cmd/server/fink/topics/{spike,notificationmgmt,conversationmgmt}.go and applied by the upsert_kafka_topics job. Locally it's always 1, so a single broker restart does interrupt you — that's an artifact of local dev, not how staging/prod behave.
Read / watch this next

Confluent Developer — Kafka Data Replication Protocol

The clearest walkthrough of leaders, followers, ISR, and leader election. Pair with the Confluent replication design doc for exact wording.

→ developer.confluent.io/courses/architecture/data-replication
→ docs.confluent.io/kafka/design/replication.html

Check yourself (from memory)

Q1. For one partition, reads and writes are served by…

One leader per partition handles all traffic; followers only fetch to stay in sync. (Newer Kafka can do follower reads, but the default and the mental model is leader-only.)

Q2. When a leader dies, the new leader is chosen from…

Only ISR members have all committed data, so only they can be promoted without losing writes. A lagging follower is ineligible.

Q3. With replication factor 3, how many brokers can you lose?

Replication factor N tolerates N−1 losses. With 3 copies, two brokers can go and the partition stays available.
Connect acks=all to the ISR in one sentence.
recall, then click to reveal
acks=all waits until every in-sync replica has the write before confirming it — so if the leader dies right after, a caught-up follower (an ISR member) becomes leader with your data intact. Pair it with min.insync.replicas ≥ 2 for real single-broker-loss safety.
Curious what "unclean leader election" is, or how a follower rejoins the ISR? Ask me — both are common senior-level follow-ups.

1. Confluent — Kafka Replication; Confluent Developer — Data Replication Protocol.