Lesson 5 · Reliability core

Delivery semantics

At-most, at-least, exactly-once — and which one your consumers actually get.

Your win: explain the three delivery semantics, show that the only thing that decides between at-most and at-least-once is when you commit the offset, and identify which one each of your consumers gets.

It all comes down to commit timing

A consumer processes an event and, at some point, commits its offset (Lesson 2–3). Whether you commit before or after processing is the whole story of delivery semantics.¹

commit BEFORE process commit AFTER process ───────────────────── ──────────────────── read → commit → process read → process → commit ↑ crash here ↑ crash here offset moved, work lost offset NOT moved, work re-run = AT-MOST-ONCE = AT-LEAST-ONCE (may lose, never dup) (never lose, may dup)

At-most-once

Commit first. A crash mid-processing loses that event. Never duplicated.

At-least-once

Commit last. A crash re-delivers the event on restart. Never lost, may duplicate. Kafka's default.

Exactly-once

Effect happens once, via idempotent producer + transactions. Powerful, heavier.

Both patterns live in your wrapper

Our internal/golibs/kafka ships two consumer implementations, and a service picks one with the AlwaysCommit() option:

internal/golibs/kafka/consumer.go

// STRICT (default) — commit AFTER the handler succeeds  →  at-least-once
strictConsumer: FetchMessage → run handler → CommitMessages()   // consumer.go:203

// LOOSE — ReadMessage auto-advances the offset regardless  →  at-most-once for a fail
looseConsumer: ReadMessage (auto-commit) → run handler   // consumer.go:165

Who uses which — and why notification's async & system consumers = strict (at-least-once). Losing a notification is bad, so re-processing on crash is the right default.
spike email & conversationmgmt = AlwaysCommit(). They accept "the offset moves regardless" to guarantee the partition never stalls — trading a rare lost message for liveness.

The dark side of at-least-once: the poison message

Strict/at-least-once has a sharp edge. If one event always fails the handler, the offset never advances — so that partition is stuck, re-delivering the same bad event forever and blocking everything behind it. This is the classic "poison message" problem, and our repo has no dead-letter topic to catch it (Lesson 6).²

That's exactly why the loose consumers exist: AlwaysCommit() can't get stuck, because the offset moves whether or not the handler succeeded. Two honest answers to the same tension — pick your poison, literally.

Where's exactly-once? We don't use it.

Kafka can do exactly-once (EOS) via an idempotent producer plus transactions.¹ Our services deliberately don't. Instead they take at-least-once and make the handler idempotent, so a duplicate has no extra effect — the same guarantee, far less machinery. That's the entire subject of Lesson 6.

Confluent — Message Delivery Guarantees

The authoritative treatment of the three semantics and how commit timing produces each. Short and precise.

→ docs.confluent.io/kafka/design/delivery-semantics.html

Check yourself (from memory)

Q1. Committing the offset after processing gives you…

Commit-after → a crash before commit re-delivers the event → never lost, possibly duplicated = at-least-once. Our strict consumer.

Q2. Why does spike use AlwaysCommit() instead of the strict mode?

The offset advances regardless of handler outcome, so a poison message can't block the partition — at the cost of possibly dropping that one failed message. A liveness-vs-loss trade.

Q3. Our services get exactly-once effect by…

No Kafka transactions here. At-least-once delivery + an idempotent handler = duplicates do nothing = exactly-once effect. Lesson 6.

One event always fails the handler in a strict (at-least-once) consumer. What happens?

recall, then click to reveal

The offset never advances, so the partition is stuck — re-delivering that poison message forever and blocking everything behind it. With no dead-letter topic (our case), the fix is either an idempotency/skip path or the loose AlwaysCommit() mode that advances regardless.

Want the exact segmentio calls (FetchMessage+CommitMessages vs ReadMessage) that make each mode behave this way? Ask me.

1. Confluent — Message Delivery Guarantees.

2. Repo: repo-kafka-map.md — commit modes & the missing DLQ (consumer.go).