Lesson 10 · Scaling, scheduling & reliability

Scheduling

Steering where Pods land — labels, affinity, and taints — and how this repo saves money with spot nodes.

Your win: explain how the scheduler decides node placement, the difference between node affinity (attract), taints/tolerations (repel), and pod anti-affinity (spread) — and how this repo uses them to run on cheap spot nodes.

Labels & selectors — the glue you've been using all along

A label is a key-value tag on an object (app: bob); a selector is a query that matches labels. You've already relied on them: a Service finds its Pods (Lesson 5), and a ReplicaSet owns its Pods (Lesson 4), by selector.¹ Scheduling reuses the same idea, but for matching Pods to nodes.

Three tools to control placement

The scheduler (Lesson 2) picks a node for each Pod based on resource requests (Lesson 8) plus these constraints:

Tool	Direction	Effect
node affinity / `nodeSelector`	Pod → node	attract: "run me on nodes with label X"²
taint + toleration	node ⟂ Pod	repel: a tainted node rejects Pods unless they tolerate the taint³
pod anti-affinity	Pod ⟂ Pod	spread: "don't co-locate my replicas on one node"

Affinity vs taints — the pairing They're complementary. Affinity is the Pod saying "I want that kind of node." A taint is the node saying "keep off unless you tolerate me." Special node pools (GPU, spot, on-demand) are usually tainted so only Pods that explicitly opt in (via a toleration) land there — and those Pods also carry affinity so they're attracted to that pool. Belt and braces.

How this repo schedules — spot nodes by default

Anchor — the default Deployment targets spot nodes The base template is literally named _util.deployment.spot (deployments/helm/libs/util/templates/_deployment.tpl:30) — by default, backend Pods run on GKE spot nodes (cheap, pre-emptible VMs). This is real cost engineering, expressed through scheduling. It works via:

Tolerations (_tolerations.v4.tpl) with flags spot, uatPool, onDemandPool — so Pods tolerate the taint on the spot node pool (and, on prod, an on-demand pool).
Affinity (_affinity.v6.tpl) — both nodeAffinity (attract to the right pool) and podAntiAffinity (spread replicas across nodes so one node loss doesn't take them all).

Spot nodes can vanish — hence the on-demand override Spot VMs can be reclaimed by the cloud at any time, so running only on spot is risky for critical paths. The library generates a sibling on-demand-node Deployment via a merge-override (_deployment.tpl:125, _util.deployment.ondemand.override) for prod, so some replicas sit on stable on-demand nodes. Notice how scheduling (this lesson), disruption tolerance (Lesson 11), and cost all interact — that's the real-world reasoning interviewers probe.

Kubernetes docs — assigning Pods to Nodes & taints/tolerations

nodeSelector, node/pod affinity and anti-affinity, and how taints repel Pods that lack a matching toleration.

→ kubernetes.io — Assigning Pods to Nodes
→ kubernetes.io — Taints and Tolerations

Check yourself (from memory)

Q1. A taint on a node…

Taints repel; a Pod needs a matching toleration to land. Affinity is the attract side.

Q2. To stop all replicas landing on one node, use…

Pod anti-affinity spreads replicas so a single node failure doesn't take them all — the repo uses it.

Q3. Backend Pods here run by default on…

The base template is _util.deployment.spot; an on-demand override adds stable replicas for prod reliability.

Node affinity vs taints vs pod anti-affinity — and this repo's spot-node setup.

recall, then click to reveal

LABELS + SELECTORS match objects (Services/ReplicaSets already use them; scheduling matches Pods↔nodes). PLACEMENT TOOLS: NODE AFFINITY / nodeSelector — ATTRACT a Pod to nodes with a label; TAINT + TOLERATION — a node REPELS Pods unless they tolerate it (special pools are tainted; opt-in Pods carry the toleration); POD ANTI-AFFINITY — SPREAD replicas across nodes. REPO: default template is _util.deployment.spot — backend runs on cheap GKE SPOT nodes via tolerations (_tolerations.v4.tpl: spot/uatPool/onDemandPool) + affinity (_affinity.v6.tpl: nodeAffinity + podAntiAffinity). Spot VMs can be reclaimed, so a sibling ON-DEMAND-NODE Deployment (_deployment.tpl:125) adds stable replicas on prod.

Want the difference between requiredDuringScheduling… and preferred… affinity, or how topology-spread constraints differ from anti-affinity? Ask me.

1. Kubernetes — Labels and Selectors.

2. Kubernetes — Assigning Pods to Nodes.

3. Kubernetes — Taints and Tolerations.