Lesson 11 · Scaling, scheduling & reliability

Disruptions & the pod lifecycle

Staying available while nodes drain and pods die gracefully — the reliability layer.

Your win: distinguish voluntary vs involuntary disruptions, explain what a PodDisruptionBudget protects, and describe graceful termination (SIGTERM → grace period → SIGKILL) — which matters directly for the spot nodes from Lesson 10.

Two kinds of disruption

Involuntary — the unplanned: a node crashes, hardware fails, a spot VM is reclaimed (Lesson 10!). You can't prevent these; you design replicas + anti-affinity to survive them.
Voluntary — the planned: draining a node for an upgrade, a rollout, scaling down. These you can pace.¹

PodDisruptionBudget — a floor under voluntary disruptions

A PodDisruptionBudget (PDB) caps how many Pods of an app may be voluntarily down at once. You set minAvailable (keep at least N up) or maxUnavailable (take at most N down). When something tries to evict Pods — a node drain — the eviction API honours the PDB and evicts only as fast as the budget allows.¹

What a PDB does and doesn't do A PDB only constrains voluntary disruptions (evictions/drains). It does not protect against a node crashing — that's involuntary. And it can block a drain: if honouring the budget means no Pod can be evicted, the drain waits. Powerful for availability, but a mis-set PDB can stall cluster maintenance.

Anchor — a PDB for every service The library generates one from deployments/helm/libs/util/templates/_pdb.tpl:4 (apiVersion: policy/v1), with maxUnavailable: {{ default 1 .Values.pdbMaxUnavailable }} (:10) — so by default at most one Pod is voluntarily down at a time. This pairs with the spot-node story (Lesson 10): when the cluster autoscaler drains a node, the PDB keeps enough replicas serving. Exception: tom disables its PDB (backend/tom/values.yaml:48-49) — a deliberate per-service choice.

The pod lifecycle — dying gracefully

When a Pod is deleted (rollout, scale-down, drain), it isn't killed instantly:²

delete Pod │ ├─ removed from Service endpoints (readiness/endpoints stop new traffic) ├─ preStop hook runs (if any) ├─ SIGTERM sent to the container ── app should finish in-flight work, stop cleanly │ … up to terminationGracePeriodSeconds … └─ SIGKILL if still alive at the deadline

The contract: your app should catch SIGTERM, stop accepting new work, drain in-flight requests/messages, and exit before the grace period ends. This is why graceful shutdown in your Go server matters — and why it interacts with spot nodes, which give a short termination notice before reclaim.

Anchor — a longer grace period for consumers Most Pods use the default grace period, but the notification team's Kafka-consumer Deployment sets terminationGracePeriodSeconds: 60 (backend/notificationmgmt/templates/deployment-consumers.yaml:36) — extra time so a consumer can finish processing and commit offsets before it's killed (your Kafka course's at-least-once + graceful-drain concern, expressed in K8s).

Kubernetes docs — Disruptions & Pod lifecycle

Voluntary vs involuntary disruptions and PDBs, then the termination sequence (SIGTERM/grace/SIGKILL, preStop).

→ kubernetes.io — Disruptions (configure a PDB)
→ kubernetes.io — Pod Lifecycle

Check yourself (from memory)

Q1. A PodDisruptionBudget limits disruptions that are…

PDBs pace voluntary evictions (drains/rollouts). A crashed node is involuntary — replicas + anti-affinity cover that.

Q2. On deletion, a container first receives…

SIGTERM (+ optional preStop) → up to terminationGracePeriodSeconds → SIGKILL. Catch SIGTERM to drain cleanly.

Q3. Notification's consumers set a 60s grace period to…

Extra drain time so an in-flight message finishes and offsets commit before SIGKILL — clean at-least-once shutdown.

Voluntary vs involuntary disruption, what a PDB does, and graceful termination.

recall, then click to reveal

DISRUPTIONS: INVOLUNTARY (node crash, spot reclaim — unpreventable; survive with replicas + anti-affinity) vs VOLUNTARY (drain, rollout, scale-down — you can pace). A PDB caps VOLUNTARY down-pods via minAvailable/maxUnavailable; the eviction API honours it during drains — but it does NOT stop crashes, and can BLOCK a drain if mis-set. Repo: _pdb.tpl:4 (policy/v1) default maxUnavailable: 1; tom disables its PDB. TERMINATION: delete → removed from Service endpoints → preStop → SIGTERM → up to terminationGracePeriodSeconds → SIGKILL. App must catch SIGTERM and drain. Repo: notification consumers use 60s grace to finish + commit Kafka offsets.

Want to see how kubectl drain interacts with a PDB, or why spot-node reclaim only gives ~30s notice? Ask me.

1. Kubernetes — Disruptions; Configure a PDB.

2. Kubernetes — Pod Lifecycle (termination).