Lesson 13 · Senior Kafka practice

Rebalances, lag, and consumer backpressure

How to reason about stuck partitions, slow consumers, group churn, and what lag is really telling you.

Your win: explain what lag really means, what rebalances cost, and how a senior engineer distinguishes “Kafka is broken” from “the consumers are overloaded or blocked.”

In plain English Plain English: lag usually means work is arriving faster than this group can safely finish it, or one partition is stuck behind something slow or failing.

Why this topic matters

Teams often talk about lag as though it were the problem. It is not. Lag is the visible symptom that tells you progress is falling behind arrival. The real job is to figure out why.

This is why senior Kafka answers sound different from beginner ones. A beginner says, “lag means the consumer is behind.” A senior says, “lag means offsets are not advancing fast enough, and now I need to distinguish under-capacity, handler slowness, downstream failure, partition blockage, or group churn.”

The lag rule Lag is not “Kafka is broken.” Lag means a producer/consumer balance, partition bottleneck, or handler-speed problem is expressing itself in offsets.

What rebalances really cost

Earlier lessons taught what a rebalance is. The deeper lesson is why rebalances are expensive even when they work correctly. While partitions are being reassigned, useful work pauses. If members churn too often, the group spends more time coordinating than progressing.

That is why consumer stability is part of throughput. A flapping or wedged consumer is not just one unhealthy pod. It is a repeated tax on the whole group.

Backend use case In this repo, health checks and self-healing loops are part of Kafka reliability because they reduce the chance that a dead or wedged consumer leaves partitions unread for long.

Common mistake Treating lag as only a scaling problem when the real issue is one blocked partition, one slow downstream dependency, or repeated group churn.

Review groups and failure behavior together

The strongest understanding of lag comes from combining group mechanics with the repo’s real failure behavior.

→ docs/kafka/lessons/0003-consumer-groups.html
→ docs/kafka/lessons/0011-failure-modes.html

Check yourself (from memory)

Q1. What is the strongest senior interpretation of consumer lag?

Lag points to a pressure or progress problem; the next step is diagnosing which one.

Why are repeated rebalances expensive even if Kafka recovers correctly?

recall, then click to reveal

Because progress pauses while partitions are reassigned, so a flapping or unstable group loses throughput to coordination overhead and delayed work.

Want a compact “lag triage” checklist for Kafka interviews and debugging? Ask me.

Sources. Confluent consumer-group docs; repo lessons and map.