Lesson 12 · Run the domain — delivery & schedules

The whole picture

Seven add-ons look like seven unrelated topics — until one real feature threads through every single one of them. Here's that thread, and the course in one view.

Your win: narrate one scheduled notification touching all seven topics, and carry the whole toolkit into an interview.

One flow, all seven topics

Follow a scheduled bulk notification — "remind these students at 9am" — and watch every add-on show up:

┌ Part 3 · run ───────────────────────────────────────────────────────────────┐ │ CRON (k8s CronJob, schedule:) fires the binary ─▶ publishes to Kafka │ │ │ │ │ KEDA gjob CONSUMER pod (scaled on lag) consumes the async-notification │ │ │ │ │ ┌ Part 2 · keep correct ─────────────────────────────────────────────────┐ │ │ │ IDEMPOTENCY: seen this id? drop. RETRIES: transient? redeliver/loop. │ │ │ └─────────────────────────────────────────────────────────────────────────┘ │ │ │ fan-out │ │ PUSH ─▶ FCM multicast (500-chunks, swallow failures) │ │ EMAIL ─▶ spike gRPC ─▶ publish-after-commit ─▶ SendGrid ─▶ webhook │ └───────────────────────────────────────────────────────────────────────────────┘ ┌ Part 1 · ship it (underneath all of the above) ───────────────────────────────┐ │ the tables came from MIGRATIONS · the code passed (most) LINT/HOOKS · │ │ the SendGrid/FCM keys were SOPS-decrypted in-process · │ │ it's all ONE BINARY/IMAGE built via Skaffold │ └───────────────────────────────────────────────────────────────────────────────┘

That's the point of a "Tier-3" tier: none of it is the feature, but the feature can't ship, stay correct, or run without all of it.

The course in one view

Part	Topic	The one thing
1 · Ship it	Migrations	forward-only `.up.sql`; fix-forward, no rollback; `gen-db-schema` feeds sqlc
	Linting & githooks	an enable-list; a bespoke pgx `sqlclosecheck` that isn't actually enforced; `LT-`/gitleaks hooks
	Secrets	SOPS encrypts values, KMS wraps the key, services decrypt in-process; a 2nd AES layer for DB keys
	Build / Skaffold	one binary, all services, chosen by a subcommand; dev copies, prod compiles; Skaffold+kind vs compose
2 · Keep it correct	Idempotency	at-least-once → record processed IDs; write the marker only after success/non-retryable
	Retries	`(retryable, err)`; in-process first, then broker; non-retryable when the effect already happened
	Outbox	same-tx insert kills the dual-write; relayed by Debezium; eureka-only + flagged
	Put together	your services publish-after-commit with status compensation — a deliberate trade-off
3 · Run the domain	Email	a status machine CREATED→QUEUED→PROCESSED + webhook; lane ⟂ SendGrid client
	Push	token multicast in 500s; retry transient, clean up unregistered, swallow the rest
	Cron	jobs are subcommands; k8s CronJob schedules; three runtime shapes, one binary
	Whole picture	this lesson — the seven interlock in every real flow

How this ties your whole track together The outbox's relay is Debezium (your CDC course). Retries re-inject JWT claims so tenancy survives (your Auth + NATS courses). Migrations stamp RLS (Auth) and feed sqlc (v2 architecture). Idempotency and the delivery pipelines run on Kafka + NATS (your eventing courses). This tier is the connective tissue — the course where everything you've learned shows up doing a day job.

Interview lightning round

🎯 Seven answers you now own

Migrations? Forward-only golang-migrate; fix-forward, no down files; a generated dump feeds sqlc.
Linting? golangci-lint enable-list; even a hand-written analyzer for unclosed pgx rows (self-test only).
Secrets? SOPS values + KMS-wrapped key, decrypted in-process; a second AES layer for DB keys.
Build? One binary, all services, chosen by a subcommand; Skaffold+kind or docker-compose locally.
Reliability? Idempotent consumers + retries + outbox-or-compensation — one per at-least-once failure.
Delivery? Email is a status machine ending at a SendGrid webhook; push is an FCM multicast that swallows failures.
Scheduling? Jobs are subcommands; k8s CronJobs schedule; a KEDA pod scales consumers on Kafka lag.

The reference shelf

Everything compressed for reuse — skim before an interview.

→ Cheat sheet · Glossary · Repo toolkit map
→ external: golang-migrate · SOPS · Outbox · Skaffold

Check yourself (from memory)

Q1. In the scheduled-notification flow, what fires the work?

A CronJob (schedule in Helm values) runs the binary; the KEDA consumer pod then processes what it publishes.

Q2. Which two add-ons guard the consumer once the message arrives?

At-least-once delivery means duplicates and transient failures — idempotency drops dupes, retries handle the transient ones.

Q3. A single notification can fan out to…

notification pushes via its own FCM path and requests email from spike over gRPC — both channels from one notification.

Recall: the one flow that touches all seven add-ons.

cron → consumer → guards → fan-out → underneath, then reveal

Scheduled bulk notification: a k8s CronJob fires the binary → publishes to Kafka → the KEDA-scaled gjob consumer pod consumes → guarded by idempotency (drop dupes) + retries (transient) → fans out to push (FCM multicast, swallow failures) and email (via spike gRPC → publish-after-commit → SendGrid → webhook). Underneath: tables from migrations, code past lint/hooks, SendGrid/FCM keys SOPS-decrypted in-process, all one binary/image (Skaffold). Seven add-ons, one feature. Ties to CDC (outbox relay), Auth/NATS (tenancy on retry), v2 (sqlc from migrations).

That's the course — all twelve lessons, all seven add-ons. You can now ship a change, keep it correct under at-least-once delivery, and run the delivery + scheduled paths of your own services. The one thing left is a retention check: ask me to run a cold mock interview across this course (and the track) and I'll write your first real learning records. Ask me anything — that's what I'm here for.

1. Synthesis of Lessons 1–11; anchors in repo-toolkit-map. Real scheduled path: cmd/server/notificationmgmt/cronjob_exec_scheduled_notification_bulk_job.go → async-notification consumer → push (FCM) + email (spike gRPC).