Lesson 9 · Applied to your services

Trace one event end-to-end

Every concept from Lessons 1–8, walked through a single email.

Your win: narrate the complete journey of one email event through spike — naming the Kafka concept at every hop. This is the "walk me through how your system uses Kafka" interview answer, and it's pure retrieval across the whole course.

How to use this lesson Read each hop, then cover it and try to say the next one and its concept from memory. If a hop is fuzzy, its lesson link is right there — go refresh, come back.

The eight hops

① client calls SendEmail (gRPC) │ ② TopicByPriority → picks communication.email-sending[.high-priority] │ ③ TracedPublishContext(topic, key=EmailID, json) ── + context headers │ ④ Hash(key) → partition acks=RequireOne → leader stores it │ ⑤ [ topic partition, durable append-only log ] │ ⑥ spike consumer group reads (AlwaysCommit, high lane first) │ ⑦ SendEmailHandler.Handle: json.Unmarshal → SendGrid │ ⑧ update DB status; return (retryable, err)

① The request arrives

A client calls spike's SendEmail gRPC endpoint (internal/spike/.../grpc/email_modifier_send_email.go:47). The handler will not send the email — it will publish an event and return. → Lesson 1: decoupling

② Choose the lane

TopicByPriority(resolvePriority(clientID)) picks either communication.email-sending or its .high-priority sibling. Two topics, on purpose. → Lesson 10: priority lanes

③ Publish — with the key and the context

TracedPublishContext(ctx, span, topic, []byte(EmailID), json). Two things ride along besides the payload:

④ Partition & durability

The Hash balancer hashes EmailID → a specific partition, so every event for this email is ordered together. The producer uses acks=RequireOne: the partition leader stores it, then the publish returns. → Lesson 4: acks · Lesson 7: leader

⑤ The event now lives in the log

It's a durable, append-only record in that partition, and it will stay there for the topic's retention window whether or not it's read. → Lesson 8: retention

⑥ The consumer group reads it

spike's SendEmailConsumer runs in group spike.consumer-group.communication.email-sending, and it starts the high-priority lane before the normal one (.../kafka/send_email_consumer.go:21,42). It uses AlwaysCommit() — the offset advances regardless, so the partition can't get wedged. → Lesson 3: groups · Lesson 5: commit mode

On the way in, TraceCarrierAndClaimInfoFromMessageHeaders rebuilds the user claims and trace context from the headers (utils.go:28), so the handler runs with the same identity the producer had. That's how auth and tracing survive a hop through Kafka.

⑦ Do the real work

SendEmailHandler.Handle does json.Unmarshal into a SendEmailEvent, then actually calls SendGrid (.../consumers/send_email_handler.go:46). This is the slow work Lesson 1 moved off the request path.

⑧ Record the outcome, signal retry-or-not

It updates the email's DB status (PROCESSED / PROCESSED_FAILED) and returns (retryable, err): SendGrid down → (true,…) retry; ErrPayloadTooLarge → (false,…) give up. Duplicates from a retry are made safe by design. → Lesson 6: retries & idempotency

The whole course in one sentence A keyed event is produced with acks=1 to a partition leader, lands durably in the log, is read by a consumer group in AlwaysCommit mode, processed by an idempotent handler that signals retry-or-drop — carrying its caller's identity and trace the entire way. That's spike's email flow, and it's Lessons 1–8 in motion.
Do this next (hands-on)

Kafka 101 — Hands-on: Producer & Consumer

Produce and consume from a topic yourself; watching offsets move makes this trace concrete. Keep repo-kafka-map.md open as the ground-truth companion.

→ developer.confluent.io/courses/apache-kafka/get-started-hands-on

Check yourself (from memory)

Q1. When SendEmail returns success to the client, the email has been…

The API returns once the event is durably appended (leader stored it, acks=1). SendGrid happens later, in the consumer.

Q2. How does the handler know who triggered the email?

The producer attaches Kafka-User-ID / Kafka-Resource-Path headers; the consumer rebuilds the claims from them, so identity survives the hop through Kafka.
Narrate the full email trace in one breath, naming a Kafka concept at each hop.
say it out loud, then click
gRPC publishes (decoupling) a keyed event (partition + ordering) with acks=1 (durability) plus identity/trace headers; it lands durably in the log (retention); spike's consumer group reads it, high lane first (groups + priority lanes) in AlwaysCommit mode (at-most-once-on-failure); an idempotent handler sends via SendGrid and returns retryable-or-not (retries). Teaching it aloud is the strongest signal you've truly got it.
Pick any hop and I'll drill deeper — e.g. what exactly is in a SendEmailEvent, or how the trace span links producer and consumer in your APM. Ask me.

Ground truth: repo-kafka-map.md — the spike email flow with file:line references.