Lesson 10 · Run the domain — delivery & schedules · your service

Push delivery (notification / FCM)

Your other delivery channel — a notification becomes device tokens, a multicast to Firebase, and a very deliberate choice to swallow the failures that email would retry.

Your win: trace a push from event to FCM, and explain the three failure choices that make push behave differently from email.

Two front doors, one handler

A push can arrive two ways, and both converge on the same domain handler:

NATS Notification.Created ──▶ HandleMessage ─┐ ├─▶ ProcessPushNotification ─▶ pushNotificationToUsers Kafka AsyncNotificationUpserting ──▶ consumer ─┘

The NATS subscriber (push_notification.go) and the Kafka async-notification consumer both route push_notification sending-methods into ProcessPushNotification. (The system is mid-migration NATS→Kafka, which is why both exist — the flag Communication_Notification_Use_Kafka_Instead_Of_NATs.)¹

Notification → tokens → multicast

The service resolves the target users to device tokens, then hands them to a Firebase wrapper that multicasts:

internal/notification/infra/push_notification_service.go:82-117 (condensed)

tokens := // device tokens where AllowNotification && token != ""
if len(tokens) == 0 { return 0, 0, nil }
switch {
case len(tokens) == 1: notificationPusher.Push(ctx, msg, tokens[0])
default:            notificationPusher.PushMultiple(ctx, msg, tokens)   // multicast
}

The Firebase wrapper lives in golibs — the service never touches the FCM SDK directly. PushMultiple chunks the tokens into batches of 500 (FCM's multicast limit) and calls SendEachForMulticast, returning a per-token success/failure result:

internal/golibs/firebase/messaging.go:18,66-72

const fcmBatchLimit = 500
// PushMultiple: chunk tokens by 500 → SendEachForMulticast → per-token PushTokenResult[]

The three failure choices — this is where push differs from email

Push handles failure by kind, not by blanket retry

Transient token failures (429/500/503/timeout) → retried in-process: retryFailedTokens re-sends just the retryable subset, up to maxAppRetries, with backoff. If a later round resolves them all, the earlier error is cleared.
Unregistered tokens (the app was uninstalled) → not a failure: they're published to a cleanup topic (InvalidDeviceTokenTopic) for async deletion, and counted as success. A dead token is a data-cleanup job, not a delivery error.
The Kafka consumer swallows the overall failure — it returns non-retryable so a broker redelivery won't re-push to the tokens that already succeeded. Re-pushing the whole batch to avoid one failed token would spam everyone else.

Contrast Lesson 9: email's per-message retry is safe because each email is one recipient; a push is a fan-out to many tokens, so blanket retry double-notifies. Same at-least-once world, opposite retry instinct — driven by the shape of the side effect.²

The links you already know

Two connections tie this back to your other services and courses:

conversationmgmt → push: a chat event is published as Notification.Created (your NATS course's flagship subject); notification consumes it and pushes. Total decoupling through one subject.
notification → email: notification doesn't send email itself — it requests it from spike over gRPC (EmailModifierServiceClient.SendEmail). So a single notification can fan out to both a push (its own FCM path) and an email (spike's pipeline from Lesson 9).¹

FCM — the Admin SDK & multicast

The provider side: SendEachForMulticast, the 500-token limit, and the BatchResponse with per-token success/failure that the golibs wrapper maps.

→ Firebase — send with the Admin SDK
→ in-repo internal/notification/infra/push_notification_service.go, internal/golibs/firebase/messaging.go

Check yourself (from memory)

Q1. How many device tokens fit in one FCM multicast?

fcmBatchLimit = 500; PushMultiple chunks a longer token list and calls SendEachForMulticast per chunk.

Q2. An unregistered device token comes back from FCM. What happens?

A dead token isn't a delivery failure — it's published to a cleanup topic for deletion and counted as success.

Q3. Why does the Kafka push consumer return non-retryable on failure?

A push fans out to many tokens; re-pushing the whole batch to fix one failure spams everyone. Transient tokens are retried in-process instead.

Recall: the push path + the three failure choices.

ingress + tokens + multicast + failures + links, then reveal

Ingress: NATS Notification.Created + Kafka async-notification → both → ProcessPushNotification → pushNotificationToUsers. Tokens → multicast: collect device tokens (AllowNotification, non-empty) → golibs NotificationPusher.PushMultiple chunks by 500 → FCM SendEachForMulticast (per-token results). Three failure choices: (1) transient token errors → in-process retry of the retryable subset (retryFailedTokens, maxAppRetries); (2) unregistered tokens → cleanup topic + counted success (not a failure); (3) Kafka consumer returns non-retryable (a broker redelivery would re-push to already-succeeded tokens). Push = fan-out → opposite retry instinct from email. Links: conversationmgmt publishes Notification.Created; notification requests email via spike gRPC.

🎯 Interview one-liner "How do you send push notifications?" → "We resolve users to device tokens and multicast through an FCM wrapper in 500-token batches, with per-token results. Failure handling is by kind: transient token errors retry in-process, unregistered tokens are cleaned up rather than failed, and we don't let the broker re-deliver — a push fans out to many tokens, so blanket retry would re-notify everyone who already got it."

Next: the last piece of "how it runs" — the cron / scheduled-jobs framework, and the three runtime shapes of the one binary. Ask me why push and email retry so differently — it's a great illustration of the side-effect shape driving the policy.

1. In-repo (verified): .../transports/nats/push_notification.go, .../transports/kafka/consumer_upsert_async_notification.go:185-196; .../services/notification_modifier_request_send_email.go:130 (spike gRPC); conversationmgmt .../infrastructure/nats/notification.go:77.

2. In-repo (verified): internal/notification/infra/push_notification_service.go:82-156 (retry subset, unregistered cleanup); internal/golibs/firebase/messaging.go:18,66-72 (500-chunk multicast); .../services/notification_modifier_push_notification.go:31-38 (Kafka swallow).