Lesson 12 · Senior Kafka practice

Partition strategy, hotspots, and parallelism trade-offs

How partition count, key choice, and uneven traffic shape throughput, ordering, and operational pain.

Your win: explain how to choose partition strategy from the workload, not from a default number, and how hotspots appear when one key or one partition attracts too much traffic.

In plain English Plain English: partitions buy you parallelism, but they also decide where ordering holds and where bottlenecks can form.

Why this page matters more than it first looks

Earlier lessons taught the mechanics cleanly: a topic is split into partitions, a key influences where a record lands, and consumer groups only parallelize up to the partition count. That is the beginner-to-intermediate reading.

The senior reading is different. Partition count is not just a Kafka setting. It is a hidden product decision about how much concurrency the workload can use, how much ordering the domain truly needs, and how much operational overhead the team is willing to absorb later.

That is why this topic matters more than it first looks. A bad partition strategy may still work in development. It starts hurting only when traffic becomes uneven, one tenant dominates the stream, or a consumer group needs more parallelism than the topic shape allows.

The partition rule Partition count is a trade-off between throughput, ordering, consumer parallelism, and coordination cost — not a number you pick once because it sounds scalable.

What a hotspot really is

A hotspot is not a mysterious Kafka bug. It is the place where your workload shape becomes visible in the partition layout. If one tenant, one aggregate id, or one customer creates a disproportionate amount of traffic, Kafka can distribute records exactly as instructed and still leave you with one overloaded partition.

This is the key mindset shift: Kafka hashes keys faithfully, but it cannot rescue a skewed domain model. If the same key must preserve ordering, then that key may also become the center of pressure. The senior question is whether that is necessary for correctness or whether the domain could tolerate a different keying rule.

Backend use case In this repo, the EmailID key keeps related email events ordered, but the broader lesson is how key choice and lane choice together shape where traffic pressure accumulates.

Common mistake Assuming “more partitions” automatically fixes uneven load when the real bottleneck is one hot key or one skewed traffic source.

Revisit partitions with a senior lens

Go back to the Kafka design material and read the partition section again, but this time as a workload-shaping decision rather than a vocabulary section.

→ Apache Kafka design docs
→ Repo Kafka map

Check yourself (from memory)

Q1. What is the most senior way to think about partition count?

Partition count shapes how the whole event flow behaves, not just broker layout.

Why doesn’t “add more partitions” always solve Kafka throughput pain?

recall, then click to reveal

Because hotspots often come from skewed keys or workload shape. More partitions help only if the traffic can actually spread across them without breaking the ordering guarantees you need.

Want me to turn this into a “how would you choose partitions for this topic?” interview drill? Ask me.

Sources. Kafka design docs; repo Kafka map.