Lesson 9 · Run the domain — delivery & schedules · your service

Email delivery (spike / SendGrid)

Your email service, end to end — a request becomes a row, a Kafka message, a SendGrid send, and a webhook event, tracked by a status column at every hop.

Your win: trace an email through spike's whole pipeline and name every status it passes through — and explain why QUEUED doesn't mean "sent."

The pipeline, as a status state machine

spike models delivery as a status column that advances at each stage. Learn the machine and you've learned the service:

SendEmail ─▶ CREATED ─publish─▶ QUEUED ─consume+SendGrid─▶ PROCESSED ─webhook─▶ (events) (grpc) (tx write) (Kafka) (lane) (send) (+sg_message_id) delivered/bounce/… │ │ ▼ publish failed ▼ send failed QUEUED_FAILED PROCESSED_FAILED

SendEmail persists the row as CREATED, resolves the lane, publishes, and moves it to QUEUED (Lesson 8's publish-after-commit). A consumer picks it up, calls SendGrid, and moves it to PROCESSED — with failure branches at each hop.¹

QUEUED is not "sent" QUEUED means only "published to Kafka." The email hasn't touched SendGrid yet. PROCESSED means "SendGrid accepted it." And even that isn't "delivered" — delivery truth arrives later via the webhook (bounces, drops). Confusing "queued" with "sent" is the classic support-ticket mistake; the status column exists precisely to tell them apart.

Two orthogonal choices: which lane, which client

Two independent decisions happen per email, and conflating them is a common error:

Choice	Decided by	Picks
Priority lane	the per-email `client_id` (`resolvePriority`)	which Kafka topic — normal vs high-priority
SendGrid client	the `EmailType` (`send_email_handler`)	transactional vs notification SendGrid account

…/consumers/send_email_handler.go:80-86

switch emailEvent.Type {
case constants.EmailTypeNotification: sgClient = h.NotiSgClient   // notification account
default:                             sgClient = h.TransSgClient   // transactional account
}

So a notification-type email can still travel the normal lane, and a transactional email can go high-priority. Lane = topic (throughput); client = which SendGrid account (deliverability). They don't interact.¹ And both the normal and high-priority consumers run in the same pod, isolated only by consumer group.

The send, and its failure policy

send_email_handler.go:107-134 (condensed)

sgMessageID, err := sgClient.SendWithContext(ctx, sgEmail)
if err != nil {
    retryable := true
    if errors.Is(err, sendgrid.ErrPayloadTooLarge) { retryable = false }   // too big → never retry
    UpdateEmail(status = PROCESSED_FAILED)
    return retryable, err
}
UpdateEmail(status = PROCESSED, sg_message_id = …)                       // success
// if THIS update fails: return false — SendGrid already accepted, retrying would send twice

This is Lesson 6's retry policy made concrete: a transient send error is retryable; a too-large payload isn't; and a DB failure after a successful send is non-retryable — the code's own comment says "maybe will send email twice," so it refuses to retry.¹

The webhook closes the loop

Delivery outcomes (delivered, bounce, dropped) come back asynchronously from SendGrid to an HTTP endpoint, keyed by the custom_args (email_id, recipient_id) the send attached:

…/controller/http/email_http_service.go:45

EmailStatusReceiver(gin) ─▶ UpsertEmailEvent ─▶ table email_recipient_events

A separate read API (GetEmailDeliveryStatuses) resolves the "effective" status per email from those events (Lesson 7 of the v2 course was that very query handler). So the full truth of an email lives across two tables: the emails status column (our progress) and email_recipient_events (SendGrid's verdicts).²

SendGrid — Mail Send & the Event Webhook

The provider side: the send API's custom_args, and the webhook events (processed/delivered/bounce/dropped) that populate email_recipient_events.

→ SendGrid — Event Webhook
→ in-repo internal/spike/modules/email/ (controller/grpc, consumers, controller/http)

Check yourself (from memory)

Q1. What does the QUEUED status mean?

QUEUED = on the Kafka topic. SendGrid isn't involved until PROCESSED; actual delivery comes via the webhook.

Q2. The priority lane and the SendGrid client are chosen by…

Lane (topic) from client_id; SendGrid account from EmailType. Orthogonal — a notification email can go the normal lane.

Q3. After SendGrid accepts the email, the status-update DB write fails. Retry?

The side effect already happened. Non-retryable, or you double-send — the same rule from Lesson 6.

Recall: the spike email pipeline, the status machine, the two choices.

states + lane/client + failure + webhook, then reveal

State machine: SendEmail → CREATED (tx write) → publish → QUEUED(/_FAILED) → consume → SendGrid SendWithContext → PROCESSED(/_FAILED) + sg_message_id → webhook → email_recipient_events. QUEUED ≠ sent (just on Kafka). Two orthogonal choices: priority lane (Kafka topic) from client_id; SendGrid client (trans vs noti account) from EmailType. Both consumers run in one pod (diff groups). Failure policy: transient send → retryable; ErrPayloadTooLarge → not; post-send DB failure → not (would double-send). Webhook closes the loop with delivered/bounce/dropped, keyed by custom_args.

🎯 Interview one-liner "Walk me through sending an email." → "We persist it as CREATED, publish to a priority-routed Kafka topic (QUEUED), and a consumer calls SendGrid and marks it PROCESSED with the message ID. QUEUED just means 'on the topic' — delivery truth comes asynchronously via SendGrid's webhook into an events table. The priority lane and which SendGrid account we use are independent decisions."

Next: the other delivery channel — push notifications through FCM, where failures are handled very differently. Ask me if the two-orthogonal-choices point needs another pass.

1. In-repo (verified): .../controller/grpc/email_modifier_send_email.go:40-84, .../controller/grpc/priority.go, .../consumers/send_email_handler.go:80-135.

2. In-repo: .../controller/http/email_http_service.go:45 (webhook), .../application/commands/upsert_email_event_handler.go, .../controller/grpc/email_reader_get_email_delivery_statuses.go.