Lesson 6 · Reliability core

Idempotency & retries

How at-least-once + a smart handler ≈ exactly-once, without transactions.

Your win: explain why at-least-once forces you to handle duplicates, how an idempotent handler makes a duplicate harmless, and how our retryable bool decides retry-vs-drop when there's no dead-letter topic.

At-least-once hands you duplicates. Plan for them.

Lesson 5 showed at-least-once can re-deliver an event after a crash or a rebalance. So the same event will hit your handler twice sometimes. The fix isn't to prevent that (expensive) — it's to make processing it twice have the same effect as once. That property is idempotency, and it's the single most important consumer-side habit in Kafka.1

The equation interviewers want at-least-once delivery + idempotent handler = exactly-once effect. Same outcome as Kafka's heavyweight exactly-once, a fraction of the complexity. This is our design.

How notification actually does it

The async-notification consumer dedupes against a store keyed by an idempotency id before doing the real work — if it's seen this id, it skips:

internal/notification/transports/kafka/consumer_upsert_async_notification.go:127 
// Idempotency guard, before processing:
seen := idempotentService.GetOne(ctx, msg.IdempotentID,
                 IdempotentTypeUpsertAsyncNotificationKafka)
if seen != nil { return false, nil } // already handled → drop, no re-work

idempotentService.Insert(ctx, msg.IdempotentID, ...)  // mark handled
processNotification(ctx, msg)                            // safe now

Because the record lives in the database, it survives restarts — so a duplicate delivered after a crash still gets caught. Note there are no Kafka transactions here at all; the guarantee is entirely application-level.

The retry contract: the retryable bool

Recall our handler signature returns (retryable bool, err error) (Lesson 1's glossary term). That bool is our retry policy:

Handler returnsMeaningWrapper does
(false, nil)successcommit / advance, move on
(true, err)transient failurewait, retry up to RetryLogicAttempts (default 10)
(false, err)permanent failuregive up — drop (loose) or stall (strict)

The retry loop (wait WaitingTimeToRetryLogic, retry N times) lives at internal/golibs/kafka/consumer.go:110. spike's email handler shows the judgement call in action:

internal/spike/modules/email/application/consumers/send_email_handler.go:108 
SendGrid failed?          → return true, err   // transient — retry
ErrPayloadTooLarge?       → return false, err  // retrying can't help — don't
status update failed
  AFTER a successful send → return false, err  // avoid double-send
Design insight "Retryable" is a claim about whether trying again could succeed. A network blip: yes. An email too big for SendGrid: never — retrying just wedges the partition. Getting this classification right is what keeps an at-least-once system healthy.

The missing dead-letter topic

Many Kafka systems route give-up messages to a dead-letter topic (DLQ) for later inspection. Our repo has none. So a permanent failure is either logged-and-dropped (loose / the handleMessageStrictly path, consumer_upsert_async_notification.go:107) or it stalls the partition (strict). Knowing this gap is real, current knowledge of your own system — exactly the kind of thing a good interviewer probes.2

Read this next

Confluent — Message Delivery Guarantees (idempotency & EOS)

See how Kafka's built-in exactly-once compares to app-level idempotency, so you can argue why we chose the latter.

→ docs.confluent.io/kafka/design/delivery-semantics.html

Check yourself (from memory)

Q1. An idempotent handler is one where processing an event twice…

Same effect as once = duplicates are harmless. That's what lets at-least-once deliver exactly-once effect.

Q2. spike returns (false, err) for ErrPayloadTooLarge because…

A too-big email fails identically every time — marking it non-retryable stops it from wedging the partition. Transient errors (SendGrid down) return true instead.

Q3. When a handler permanently fails and there's no DLQ, the message is…

Loose mode logs & drops; strict mode stalls the partition on it. No DLQ exists in our repo to catch it — a known gap.
Why prefer at-least-once + idempotency over Kafka's real exactly-once?
recall, then click to reveal
Same guarantee (each event has effect once) with far less machinery — no transactional producer, no coordinator, no cross-partition atomic commit. You push the dedupe into your handler (e.g. a DB-backed idempotency key), which you control and can reason about. It's the pragmatic default for most systems.
Want to design an idempotency key for a new consumer, or add a DLQ pattern to one of our services? Ask me — great applied practice.

1. Confluent — Message Delivery Guarantees.

2. Repo: repo-kafka-map.md — idempotency, the retryable contract, no DLQ.