Lesson 8 · Keep it correct — reliability · your code

Reliability, put together

The outbox is the "correct" answer — and your own services deliberately don't use it. Here's what they do instead, why that's a defensible call, and how the three tools combine.

Your win: explain publish-after-commit with status compensation, say honestly where it's weaker than the outbox, and choose between them for a given flow.

What spike actually does: publish-after-commit

spike's SendEmail doesn't insert an outbox row. It commits the email, then publishes — two writes — and patches a status column if the publish fails:

internal/spike/modules/email/controller/grpc/email_modifier_send_email.go:47-77
// 1. CreateEmail already committed the row (status CREATED) — the tx is closed
err = svc.KafkaMgmt.TracedPublishContext(ctx, …, topic, key, payload)   // 2. publish AFTER commit
if err != nil {
    svc.EmailRepo.UpdateEmail(ctx, …, {status: "QUEUED_FAILED"})   // 3a. compensate
    return …, status.Error(codes.Internal, …)                        //     tell the caller
}
svc.EmailRepo.UpdateEmail(ctx, …, {status: "QUEUED"})            // 3b. success

This is a dual write — exactly what the outbox exists to avoid. The mitigation is a status column plus the caller: a failed publish is marked QUEUED_FAILED and reported, and ExternalBulkSendEmails returns a per-email status map so the caller retries only the ones that failed.1

The honest gap If the process crashes between the commit and the publish, nothing auto-relays that email — it sits at CREATED forever unless a caller or a sweep notices. That's the precise risk the outbox's doc-comment warns about ("Debezium can never miss it even if the process crashes after the commit"). spike accepts that risk on purpose. So does notification's async path — durability across the Kafka hop is provided by the idempotent table (Lesson 5) and caller retries, not by an outbox.

Why that's a defensible choice

Outbox vs publish-after-commit — the trade-off
Transactional outboxPublish-after-commit
lost-event risknone (atomic)real (crash-in-the-gap)
costoutbox table + Debezium connector + CDC infraa status column + a retrying caller
latency+ the CDC hopimmediate
best whenthe event MUST NOT be lost (money, grading)a caller already retries & a missed one is recoverable
eureka's assessment flow uses the outbox because a lost grading event is unacceptable. An email that occasionally needs a re-request is recoverable and has a retrying caller — so the cheaper publish-after-commit is a reasonable engineering call, not a bug. Naming it as a trade-off (not "the wrong way") is the senior move.

The three tools, combined

Reliability here isn't one pattern — it's three that cover different failures:

PUBLISH SIDE │ make the send reliable outbox (eureka) │ → atomic: never lose the event publish-after-commit │ → dual-write + status compensation + caller retry ───────────────────────────────────────────────────────── TRANSPORT │ survive redelivery retries (Lesson 6) │ → (retryable, err); in-process then broker ───────────────────────────────────────────────────────── CONSUME SIDE │ survive duplicates idempotency (Lesson 5)│ → record processed IDs, drop dupes

A message can be reliably produced (outbox or compensation), redelivered on failure (retries), and safely reprocessed (idempotency). Real systems need all three, because at-least-once delivery means duplicates and gaps are normal, not exceptional.

Read this next

Outbox vs dual-write — the trade-off

Re-read the outbox page's "problem" section with spike's publish-after-commit in mind — the gap it describes is exactly the one spike accepts.

→ microservices.io — Transactional outbox
→ in-repo .../email/controller/grpc/email_modifier_send_email.go:47-77, deployments/docs/outbox_pattern.md

Check yourself (from memory)

Q1. How does spike publish an email event reliably?

Publish-after-commit with a status column: failed publish → QUEUED_FAILED + reported to the caller, who retries. It's a dual write.

Q2. What's the risk spike accepts by not using an outbox?

The dual-write gap: a crash after commit but before publish leaves a CREATED row nobody auto-relays — recoverable via the retrying caller.

Q3. Which three tools together make async delivery reliable?

Reliable publish (outbox or compensation) + retries/redelivery + idempotency (dedup). Each covers a different failure of at-least-once delivery.
Recall: publish-after-commit, the trade-off, and the three combined tools.
what spike does + why + the trio, then reveal
spike/notification: publish-after-commit — commit the row (CREATED), then publish; on failure mark QUEUED_FAILED + report so the caller retries (bulk returns a per-email status map). It's a dual write; durability leans on status + caller + the idempotent table, NOT an outbox. Gap: crash between commit and publish strands the row (the exact risk the outbox avoids) — accepted on purpose. Trade-off: outbox = never lose the event, costs CDC infra + latency (use for money/grading); publish-after-commit = cheap + immediate, needs a retrying caller (use when a miss is recoverable). Three tools: reliable publish (outbox/compensation) + retries (redelivery) + idempotency (dedup) — each covers a different at-least-once failure.
🎯 Interview one-liner "Do you use the outbox everywhere?" → "No — it's the right tool when an event must never be lost, like grading, and there we pay for it with a Debezium relay. For recoverable flows like email, we publish-after-commit with a status column and a retrying caller — a deliberate trade-off, not an oversight. And regardless of how we publish, we retry on the transport and dedupe on the consumer, because delivery is at-least-once."
That's Part 2 — you can reason about correctness under at-least-once delivery end to end. Take the four quizzes cold, then tell me "build Part 3" for how your services actually run: email, push, and scheduled jobs. Ask me anything about the trade-off first.

1. In-repo (verified): .../email/controller/grpc/email_modifier_send_email.go:47-77 (publish-after-commit + QUEUED_FAILED); .../email_modifier_external_bulk_send_emails.go (per-email status map); contrast internal/golibs/outboxevents/outbox_events.go:89-104.