Lesson 11 · Applied to your services

Failure modes & the missing DLQ

"What happens when X fails?" — answered for every X, in your code.

Your win: for each way a Kafka consumer can fail, state exactly how our system responds — and describe the dead-letter queue we don't have and how you'd add one. This is the "how resilient is your system?" interview, end to end.

The failure table

Interviewers rarely ask "what is Kafka?" — they ask "what happens when a consumer dies mid-message?" Here is the whole matrix for our services, each row pointing back at the lesson that explains the mechanism.

What fails	What Kafka / our wrapper does	Lesson
Handler hits a transient error	returns `(true, err)` → wait `WaitingTimeToRetryLogic`, retry up to `RetryLogicAttempts` (10)	6
Handler hits a permanent error	returns `(false, err)` → strict stalls the partition; loose logs & drops	5
Same event delivered twice	idempotency guard (DB-keyed) makes the duplicate a no-op	6
Consumer goroutine/pod dies	group rebalances; survivors resume from last committed offset	3
Broker holding a leader dies	new leader elected from the ISR; consumer reconnects	7
Broker connection drops	reader retries; on failure `recreateReader()` reshuffles brokers + backs off	below
Producer send fails	writer retries (`MaxAttempts` ~∞); on give-up, email marked `QUEUED_FAILED`	4

The poison message, revisited

The nastiest case is an event that fails every time (bad data, a bug). In a strict consumer this never commits, so the partition is stuck re-delivering it forever — everything behind it blocked. Our services defuse this two ways: mark truly-unfixable errors non-retryable so they're dropped rather than looped (spike's ErrPayloadTooLarge), and use AlwaysCommit() where liveness matters more than the odd lost message (consumer_upsert_async_notification.go:107 logs & drops on non-retryable).¹

The self-healing run loop

A consumer that just died on the first network hiccup would be fragile. Our wrapper runs consumers in a loop that doesn't self-end (the preferred mode, behind the Kafka_consumer_disable_self_end flag): on a read failure it rebuilds the reader against a reshuffled broker list and backs off exponentially, rather than giving up.

internal/golibs/kafka/kafka.go:672 · consumer.go:92 · kafka.go:76

runConsumerWithoutSelfEnd:
    loop {
      err := consume()
      if err != nil {
        recreateReader()      // shuffle brokers, fresh connection
        backoff(5s … 60s)     // exponential, don't hammer
      }
    }

The liveness backstop

If a consumer goroutine dies anyway, AllConsumersRunning() flips false, the pod's /healthz returns 503, and Kubernetes restarts the pod (cmd/server/notificationmgmt/consumers.go:110) — which triggers a clean rebalance so those partitions get picked up. Kafka's fault tolerance plus a plain health check: no message silently stops being processed.

The DLQ we don't have

A dead-letter queue is a dedicated topic where give-up messages are parked for inspection instead of dropped or looped. Our repo has none — permanent failures are logged-and-dropped (loose) or stall the partition (strict).² Knowing this gap and the standard fix is a strong senior answer:

How you'd add one On non-retryable failure (or after N retries), publish the message + error to a *.dlq topic and commit the original — the partition keeps moving, and nothing is silently lost. Then a separate consumer or dashboard triages the DLQ. Confluent's guidance: don't add a DLQ until you've decided who processes it.

Confluent — Error Handling Patterns for Apache Kafka

Retry topics, dead-letter queues, and when to fail-fast vs drop vs park. Pair with Confluent's DLQ guide.

→ confluent.io/blog/error-handling-patterns-in-kafka
→ confluent.io/learn/kafka-dead-letter-queue

Check yourself (from memory)

Q1. A message fails permanently in a strict consumer with no DLQ. The partition…

Strict commits only on success, so a permanent failure never advances the offset — a stuck partition. Loose mode would drop it instead.

Q2. A consumer pod crashes. What keeps its partitions from going unread?

The group rebalances; survivors take over from the last committed offset. Our /healthz 503 also restarts a wedged pod to trigger this.

Q3. A well-designed DLQ step, on giving up, should also…

Park the message in the DLQ and commit the original, so the partition keeps moving. A DLQ that doesn't commit still stalls.

Interviewer: "A consumer dies mid-message. Walk me through what happens." Answer from memory.

say it out loud, then click

Nothing was committed for the in-flight message (strict/at-least-once), so on the group rebalance a surviving member picks up that partition from the last committed offset and re-processes it — at-least-once. The handler is idempotent, so re-processing is safe. If the pod itself wedged, the health check restarts it, which triggers the rebalance. No message is lost or silently skipped.

🎓 You've finished the course Lessons 1–11 cover the interview arc: the log & data model, groups, the reliability core, and how it all composes in your services. The last mile isn't more reading — it's retrieval under pressure. Ask me to run a mixed mock interview across all 11 lessons; where you're solid, I'll write learning records so we both know it's locked in.

Ready for the mock interview, or want to design a real DLQ for one of our consumers as hands-on practice? Ask me.

1. Repo: repo-kafka-map.md — commit modes, retry loop, self-healing run loop, health check.

2. Confluent — Error Handling Patterns for Apache Kafka; Confluent — Kafka Dead Letter Queue.