Lesson 6 · Keep it correct — reliability

Retries & redelivery

One boolean decides a message's fate. Here's what "retryable" actually triggers — and why it loops in-process before it ever falls back to the broker.

Your win: explain the (retryable, err) contract, and trace what happens on a retryable failure in both Kafka and NATS — including the in-process-first surprise.

The contract: one bool per handler

Every consumer handler — Kafka and NATS alike — returns the same pair:

internal/golibs/kafka/kafka.go:93 · internal/golibs/nats/jetstream.go:271
type MsgHandler func(ctx context.Context, data []byte) (retryable bool, err error)
// bool: whether this message should be retried

err == nil → done, acknowledge. err != nil + retryable=true → try again. err != nil + retryable=false → give up and drop (don't redeliver). The handler decides its own policy with that bool — you saw the idempotency handler use it in Lesson 5.1

The surprise: Kafka retries in-process first

You might expect retryable=true to immediately hand the message back to the broker. It doesn't. The Kafka consumer wraps the handler in the repo's retry helper and loops in the same process first:

internal/golibs/kafka/consumer.go:110-159 (condensed)
try.DoWithCtx(ctx, func(_ context.Context, attempt int) (bool, error) {
    isRetryLogic, err := handleMsg(ctx, payload)
    isCanRetry := attempt < c.option.retryLogicAttempts
    if err != nil && isCanRetry && isRetryLogic {
        time.Sleep(c.option.waitingTimeToRetryLogic)   // ← sleep + loop, in-process
        return isRetryLogic, err
    }
    return false, err
})

The generic helper try.Do/DoWithCtx caps at 10 attempts globally, and here at retryLogicAttempts. Only after in-process retries are exhausted does broker redelivery come into play — and even that depends on the consumer mode:2

Consumer modeOn unrecovered error
strictdoes not commit the offset → the broker redelivers later
loose / AlwaysCommitcommits anyway → no broker redelivery (the in-process retries were the only retries)

NATS: don't ack, and it comes back

NATS is simpler and more classic. On a retryable error the handler simply doesn't acknowledge; JetStream redelivers after the ack timeout, up to a cap:

internal/golibs/nats/jetstream.go:719-729 (condensed)
retry, err := cb(ctx, dataInMsg.Payload)
if err != nil {
    if !retry { msg.Ack() }    // non-retryable → ack (drop). retryable → do NOTHING → redeliver
    return
}
msg.Ack()                     // success → ack

The knobs are per subscription: MaxDeliver(n) (how many times) and AckWait(d) (how long before redelivery). Real example — the push-notification subscriber uses MaxDeliver(3), AckWait(90s).3

The retryable bool is a policy decision

Handlers choose retryable per failure — and it's a real design call "Retry everything" is a bug: it double-sends and it blocks the queue on poison messages.
Bonus: tenancy survives the retry On the Kafka path the consumer re-injects the JWT claims into the context before calling the handler (consumer.go:123, ContextWithJWTClaims) — the same trick your NATS course flagged. So retries and redeliveries still run under the right tenant and trace.
Read this next

The handler contract & the retry helper

Read the two MsgHandler definitions and try.Do side by side — the whole retry story is those three functions.

→ in-repo internal/golibs/kafka/{kafka.go:93,consumer.go:110-159}, internal/golibs/nats/jetstream.go:271,719-729, internal/golibs/try/try.go

Check yourself (from memory)

Q1. On a retryable Kafka error, what happens first?

try.DoWithCtx loops in-process (sleep + retry) up to the attempt cap; broker redelivery only follows if a strict consumer then leaves the offset uncommitted.

Q2. How does a NATS handler ask for redelivery?

Retryable error → don't ack → JetStream redelivers after AckWait, up to MaxDeliver. Non-retryable → ack (drop).

Q3. Why does spike return non-retryable after SendGrid accepts an email?

The side effect already happened. A retry would double-send, so the post-send failure is deliberately non-retryable. "Retry everything" is a bug.
Recall: the retry contract, in-process-first, and the per-handler policy.
contract + Kafka + NATS + policy, then reveal
Contract: (retryable bool, err error) — nil=ack; err+true=retry; err+false=drop. Kafka: retries in-process first (try.DoWithCtx, sleep+loop, cap 10 / retryLogicAttempts); only then does a strict consumer leave the offset uncommitted → broker redelivers (loose/AlwaysCommit commits regardless). NATS: retryable → don't ack → redeliver after AckWait up to MaxDeliver (push sub: 3 / 90s); non-retryable → ack (drop). Policy: transient→true; permanent(malformed)→false; already-had-effect→false (spike post-send DB fail & push failures = non-retryable to avoid double-send). Bonus: JWT claims re-injected on retry (tenancy survives).
🎯 Interview one-liner "How do retries work?" → "Every handler returns a retryable bool. We retry in-process first with a capped backoff; only if that's exhausted does a strict consumer leave the offset uncommitted for broker redelivery (NATS just doesn't ack, up to MaxDeliver). Crucially, retryable is a policy call — transient errors retry, but anything with a side effect that already happened is non-retryable, so we never double-send."
Next: the pattern that makes "update the DB and publish a message" safe in the first place — the transactional outbox. Ask me about strict vs loose consumers if the offset behaviour is fuzzy.

1. In-repo (verified): internal/golibs/kafka/kafka.go:93, internal/golibs/nats/jetstream.go:271.

2. In-repo (verified): internal/golibs/kafka/consumer.go:110-159; internal/golibs/try/try.go:11,24,41 (cap 10).

3. In-repo (verified): jetstream.go:719-729; .../transports/nats/push_notification.go:36-39 (MaxDeliver 3, AckWait 90s); spike .../consumers/send_email_handler.go:110-135; .../services/notification_modifier_push_notification.go:31-38 (push swallowed).