Lesson 8 · Reliability core

Retention vs compaction

Two ways a topic decides what to keep — and they mean opposite things.

Your win: explain the two cleanup.policy modes, bust the "Kafka deletes messages after they're read" myth, and say when you'd reach for compaction over plain retention.

A topic can't keep everything forever

The log grows without bound, so every topic has a cleanup.policy that decides what to discard. There are two modes.1

delete — retention

Drop events older than retention.ms (time) or beyond retention.bytes (size). The default. Age-based housekeeping.

compact — compaction

Keep the latest event per key; discard older values for that key. The topic becomes a snapshot of "current value per key."

Bust the myth: retention ≠ consumption

The trap Kafka does not delete a message once it's been read. Consuming never removes anything — reading only advances an offset (Lesson 2). Events age out purely by the topic's retention policy, whether zero consumers or a hundred read them. A topic with no consumers still deletes on schedule.

This is why replay works: because events linger for their full retention window regardless of who read them, a new consumer group can start from the earliest offset and reprocess history. Reading is non-destructive — that's a defining Kafka property, and a frequent interview check.

Retention in your topics

Our topics set retention explicitly in the fink topic configs:

cmd/server/fink/topics/notificationmgmt.go (and conversationmgmt.go, spike.go) 
ConfigEntries: []kafka.ConfigEntry{
    { ConfigName: "retention.ms", ConfigValue: "604800000" },  // 7 days
}
// notification/conversationmgmt topics: 7–30 days retention.
// EmailSending* topics set NO retention.ms → they inherit the broker default.
Anchor All our team's topics use the default delete policy — they're event streams (send this email, sync this membership), not state tables. None are compacted. Recognising that distinction — "is this a stream of events or a table of current state?" — is exactly how you'd decide the policy for a new topic.

When you'd choose compaction instead

Compaction turns a topic into a durable key→latest-value changelog: replay it and you rebuild current state, with no unbounded growth. Kafka itself relies on this — the internal __consumer_offsets topic (where your groups' committed offsets live, Lesson 3) is compacted, because all anyone needs is the latest offset per group/partition, not its whole history.1

Retention (delete)Compaction (compact)
Keepseverything within the windowlatest value per key
Best forevent streams, audit logscurrent-state / changelog topics
Replay givesfull history in the windowreconstructed current state
Our topicsall of them (7–30 days)none (but __consumer_offsets is)
Read this next

Confluent — Kafka Log Compaction

The authoritative description of cleanup.policy=compact, tombstones, and how a compacted topic behaves as a changelog.

→ docs.confluent.io/kafka/design/log_compaction.html

Check yourself (from memory)

Q1. When does Kafka delete a message under the default policy?

Deletion is by age/size, not by consumption. Reading is non-destructive — which is what makes replay possible.

Q2. A compacted topic guarantees it retains…

Compaction keeps the newest event for each key and discards older ones — a current-state changelog, like __consumer_offsets.

Q3. Our team's topics use plain retention because they are…

"Send this email", "sync this membership" are discrete events — you want the stream within a window, not just the latest per key.
Same topic, key = userID. Under delete vs compact, what survives after a year?
recall, then click to reveal
delete: only events inside the retention window (e.g. last 7 days) — older ones are gone regardless of key. compact: the latest event for each userID survives indefinitely, older values for that same user are removed. Streams vs current-state — pick per topic.
That's Part 2 complete — the reliability core. Want to start Part 3 (tracing one event end-to-end through spike, the priority-lane pattern, failure modes), or a mixed interview drill across Lessons 1–8? Ask me.

1. Confluent — Kafka Log Compaction; Apache Kafka docs — Log Compaction.