Lesson 11 · Run the domain — delivery & schedules

Cron / scheduled-jobs

How scheduled and background work runs — no in-process scheduler at all, just the one binary wearing three different hats.

Your win: explain how a job is registered and scheduled, and name the three runtime shapes of the single binary — and what scales each.

A "job" is a cobra subcommand

Recall the single binary (Lesson 4). Beyond gserver, any function can be exposed as a job — a subcommand under gjob — via one bootstrap hook:

internal/golibs/bootstrap/command.go:191,240

type JobFunc[T any] func(ctx context.Context, c T, rsc *Resources) error

func RegisterJob[T any](name string, s JobFunc[T]) *JobBuilder[T]   // → a cobra subcommand "name"

A service calls RegisterJob("notificationmgmt_archive_notification", fn) in an init(); the binary then dispatches by the name argument. The ctx handed to the job already handles SIGTERM, and sql_migrate from Lesson 1 is just another such job.¹

The scheduler is Kubernetes, not Go

There is no in-process cron loop The Go job entrypoints have no timer — they run once and exit. Scheduling is a Kubernetes CronJob rendered from Helm values: a schedule: cron expression runs the job binary on a timer. So "every minute" lives in YAML, not in Go.²

deployments/helm/backend/notificationmgmt/values.yaml:58,84

cronjobs:                                    # recurring → k8s CronJob
  send-scheduled-notification:
    cmd: notificationmgmt_send_scheduled_notification
    schedule: '*/1 * * * *'
  archive-notification:
    cmd: notificationmgmt_archive_notification
    schedule: '*/5 * * * *'
jobs:                                        # run-once → k8s Job (migrations, triggers)
  trigger-upsert-system-notification:
    cmd: notificationmgmt_trigger_upsert_system_notification

So jobs: = run-once, cronjobs: = recurring — but both are the same RegisterJob mechanism; the only difference is whether Helm gives them a schedule:. Locally you run one with ./local/run.bash cron -n <name>.²

A cron is often just a trigger send-scheduled-notification doesn't send anything itself — it dials the notificationmgmt gRPC service and calls SendScheduledNotification for a one-minute window. The cron is a thin, stateless kick; the real work stays in the service. That keeps the scheduled path identical to the request path.

The three runtime shapes of one binary

The same image runs in three distinct ways, and telling them apart is the whole mental model:

Shape	What it is	Scaled by
gRPC pod	`gserver` — the service + its in-pod Kafka consumers / NATS subscribers	the service Deployment (HPA)
gjob consumer pod	a long-running job (`notificationmgmt_consumers`) running only the async-notification consumer	KEDA on Kafka lag
CronJob	a short-lived job on a `schedule:` that runs to completion	the cron timer (not scaled)

The gjob consumer pod + KEDA Some Kafka consumers run inside the gRPC pod; the heavy async-notification consumer runs in its own pod (registered with RegisterJob("notificationmgmt_consumers", RunConsumers), long-running, blocking on ctx.Done()). That pod is autoscaled by KEDA on Kafka consumer-group lag (via a Prometheus query), not on CPU — when the backlog grows, KEDA adds replicas; when it drains, it scales back down. Isolating the noisy consumer lets it scale independently of the request-serving pod.³

Kubernetes CronJob & KEDA

The two scheduling/scaling primitives behind this: the CronJob timer and KEDA's lag-based autoscaling.

→ k8s — CronJob · keda.sh/docs
→ in-repo internal/golibs/bootstrap/command.go:191,240, cmd/server/notificationmgmt/{cronjob_*.go,consumers.go}, deployments/helm/backend/notificationmgmt/

Check yourself (from memory)

Q1. What actually triggers a cronjob on schedule?

The Go job runs once and exits; the schedule: in Helm values becomes a k8s CronJob that runs it on a timer.

Q2. How is the dedicated async-notification consumer pod scaled?

KEDA watches Kafka lag (via Prometheus) and scales the -consumers deployment up when the backlog grows — not on CPU.

Q3. A cronjob, a consumer pod, and a gRPC server here are…

One binary, one image; a cobra subcommand (gserver vs a gjob name) and the mounted config decide which shape it runs as.

Recall: RegisterJob, the k8s scheduler, and the three runtime shapes.

Register: bootstrap.RegisterJob[T](name, fn) → a cobra subcommand under gjob; ctx is SIGTERM-aware; sql_migrate is one too. Schedule = k8s, not Go: job entrypoints run once and exit; Helm cronjobs: (with schedule:) → k8s CronJob; jobs: = run-once. Same RegisterJob mechanism. Local: ./local/run.bash cron -n <name>. A cron is often a thin trigger (calls an RPC back into the service). Three runtime shapes, one binary: (1) gRPC pod (gserver + in-pod consumers, HPA); (2) gjob consumer pod (notificationmgmt_consumers, long-running, KEDA on Kafka lag, not CPU); (3) CronJob (short-lived, timer-driven). Subcommand + config pick the shape.

🎯 Interview one-liner "How do you run scheduled and background work?" → "Every job is a subcommand of our one binary, registered through a bootstrap helper. There's no in-process scheduler — Kubernetes CronJobs from Helm values drive the timers, and the Go entrypoint just runs once. Heavy Kafka consumers run in a dedicated pod that KEDA autoscales on consumer-group lag, so backlog spikes don't touch the request-serving pods."

Last lesson: the whole picture — how all seven add-ons interlock in one real flow, plus the course recap and interview prompts. Ask me if the three runtime shapes still blur together.

1. In-repo (verified): internal/golibs/bootstrap/command.go:191,240-254; cmd/server/notificationmgmt/cronjob_send_scheduled_notifications.go:18.

2. In-repo (verified): deployments/helm/backend/notificationmgmt/values.yaml:58-67,84-94 (cronjobs vs jobs); local/run.bash:194 (cron -n).

3. In-repo: cmd/server/notificationmgmt/consumers.go:25,32 (RegisterJob + RunConsumers); deployments/helm/backend/notificationmgmt/templates/scaledobject-consumers.yaml (KEDA on kafka_consumergroup_lag).