Lesson 2 · Foundations

Topics, partitions & offsets

Why a topic is split — and why that split is the whole game.

Your win: explain why partitions exist, what an offset is, and predict which partition one of your emails lands in — and why that guarantees its order.

A topic is not one log — it's several

In Lesson 1 we called a topic "an append-only log." The full truth: a topic is split into partitions, and each partition is its own ordered, append-only log.¹ The topic is just the name over the whole set.

topic: communication.email-sending partition 0: [e0][e1][e2][e3][e4] ─▶ append partition 1: [e0][e1][e2] ────────▶ append partition 2: [e0][e1][e2][e3] ─────▶ append ▲ offset = position within THIS partition (0,1,2,…)

Two facts fall straight out of this picture, and interviews lean on both:

Offset

An event's position within one partition — a number that only ever grows. "Where a consumer is up to" is just an offset per partition. There is no global, topic-wide position.

Ordering

Guaranteed only within a partition. Across partitions, all bets are off. So if two events must stay in order, they must land in the same partition.

Why split at all? Parallelism.

Partitions are the unit of parallelism. One partition can be read by one consumer at a time; three partitions can be read by three consumers at once.² Want to process email faster? Add partitions and consumers. (We'll see the consumer side in Lesson 3.)

The tension More partitions = more parallelism, but ordering only holds inside a partition. So you can't have both "total ordering" and "lots of parallelism" for the same stream. You choose per key — next.

The key decides the partition

When a producer appends an event with a key, a partitioner hashes the key to pick a partition. Same key → same hash → same partition → those events stay ordered relative to each other. No key → spread round the partitions for balance.

Look at what our producer does — this is why it passes EmailID as the key and configures a Hash balancer:

internal/golibs/kafka/kafka.go:465 · internal/spike/.../email_modifier_send_email.go:47

// The wrapper's writer uses a hash-of-key partitioner:
writer := &kafka.Writer{ Balancer: &kafka.Hash{}, ... }

// spike publishes each email keyed by its EmailID:
TracedPublishContext(ctx, spanName, topic,
    []byte(email.EmailID), // key → Hash → partition
    payload)

Anchor Every event for a given EmailID lands on the same partition, so retries/updates for one email stay in order — while different emails spread across partitions and get processed in parallel. That single choice buys both ordering (per email) and throughput (across emails).

The same pattern shows up in conversationmgmt: system messages are keyed by conversationID (infrastructure/kafka/message.go:22) so one conversation's messages never reorder — a very natural requirement for a chat system.

Where offsets are stored

A consumer periodically commits its offset — Kafka records it in a special internal topic, __consumer_offsets.³ That's how a restarted consumer knows where to resume. We'll unpack committing (and the at-least-once question it creates) in Lesson 3 and Lesson 5.

Confluent — Kafka Consumer Design (Consumers, Groups & Offsets)

The authoritative description of partitions, offsets, the pull model, and the __consumer_offsets topic. Read the "Offset Management" part.

→ docs.confluent.io/kafka/design/consumer-design.html

Check yourself (from memory)

Q1. Within a Kafka topic, message order is guaranteed…

Each partition is its own ordered log; there is no global order across a topic. This is the most-tested Kafka fact.

Q2. With key = EmailID and a Hash balancer, two events for the same email…

Same key → same hash → same partition, so their relative order is preserved. Different keys spread out for parallelism.

Q3. An offset identifies an event's position…

Offsets are per-partition. "Consumer progress" is one committed offset per partition, stored in __consumer_offsets.

Why are partitions the unit of both parallelism and ordering?

recall, then click to reveal

Because each partition is an independent ordered log. Different partitions can be read in parallel by different consumers (parallelism), but ordering only holds inside a single partition. So you route events that must stay ordered to the same partition via their key — trading some parallelism for order, exactly where you need it.

Want to see what happens if you add a partition to a live topic, or why "no key" behaves differently? Ask me.

1. Apache Kafka Documentation — Topics & partitions.

2. Confluent Developer — Partitions.

3. Confluent — Kafka Consumer Design (offset management, __consumer_offsets).