Lesson 9 · Scaling, scheduling & reliability

Autoscaling: HPA / VPA / KEDA

Three ways to scale — and why this repo scales consumers on Kafka lag, not CPU.

Your win: distinguish HPA (more pods), VPA (bigger pods), and KEDA (event-driven, scale-to-zero) — and explain the repo's real setup: KEDA scaling on request rate and Kafka consumer lag.

Three autoscalers, three axes

Autoscaler	Changes	Reacts to
HPA (Horizontal Pod Autoscaler)	the number of Pods	a metric target, usually CPU/memory %
VPA (Vertical Pod Autoscaler)	each Pod's size (requests/limits)	observed usage over time
KEDA (Event-Driven Autoscaling)	the number of Pods, incl. to/from zero	external events: queue depth, Kafka lag, Prometheus metrics

HPA is a control loop (Lesson 1) that adjusts a Deployment's replica count to hit a target, re-checking roughly every 15s.¹ Its blind spot: it only knows resource metrics. If your bottleneck is "10,000 messages queued," CPU may look fine while you fall behind. That's what KEDA fixes.

KEDA in one line KEDA scales a workload on external event sources — and can scale to zero when idle, then back up when work appears. Under the hood it still generates an HPA for the 1→N range; KEDA itself owns the 0↔1 activation.² So it's not a competitor to HPA — it's a smarter front-end that adds event triggers and scale-to-zero.

How this repo scales — KEDA first, HPA as fallback

Anchor — the decision logic The library orchestrator picks the autoscaler in deployments/helm/libs/util/templates/_app.tpl:39-56: if the KEDA CRD (keda.sh/v1alpha1) is installed and .Values.keda is set, use a KEDA ScaledObject; else if .Values.hpa is set, use a plain HPA. The code even comments "Keda is using HPA under the hood, so we can't also use HPA" (:52). In backend, KEDA wins — plain HPA is essentially the fallback. Meanwhile VPA is applied to nearly everything but mostly with updateMode: Off (_vpa.tpl:45) — it recommends sizes without auto-applying them.

The two KEDA triggers you'll see

Anchor — request-rate scaling util.keda.requestRateScaleObject (_keda.tpl:109) scales the main service Pods on request rate, read from a Prometheus query over Istio's istio_requests_total metric. Defaults: minReplicaCount 2, maxReplicaCount 8, poll every 30s, 180s cooldown (:121-124). So traffic → more pods, quiet → fewer (down to 2).

Anchor — Kafka-lag scaling (your team's service!) The notification team's consumers scale on Kafka consumer lag — the tie-in to your Kafka course. backend/notificationmgmt/templates/scaledobject-consumers.yaml:4 is a ScaledObject targeting the -consumers Deployment (Lesson 4); its trigger queries sum(kafka_consumergroup_lag{consumergroup="…"}) (:45) and scales up when lag crosses a threshold — 7500 in notificationmgmt/prod-tokyo-values.yaml. This is the textbook KEDA use case: "messages are piling up faster than we're consuming → add consumers," which CPU-based HPA would completely miss.

Kafka topic lag climbs past 7500 │ ▼ Prometheus scrapes kafka_consumergroup_lag │ ▼ KEDA ScaledObject → generates/updates an HPA → more -consumers Pods │ (down to 0 when idle-capable) ▼ lag drops → cooldown → scale back down

Kubernetes HPA + KEDA scaling concepts

The HPA control loop and target model, then KEDA's ScaledObject, triggers, and scale-to-zero activation phase.

→ kubernetes.io — Horizontal Pod Autoscaling
→ keda.sh — Scaling deployments (ScaledObject)

Check yourself (from memory)

Q1. HPA, VPA and KEDA respectively change…

HPA = more pods (metric), VPA = bigger pods (usage), KEDA = more pods on external events, incl. to/from zero.

Q2. KEDA can do what a plain HPA cannot?

External-event triggers + scale-to-zero. It generates an HPA for the 1→N range under the hood.

Q3. This repo scales notification consumers on…

A KEDA ScaledObject on kafka_consumergroup_lag (threshold 7500) — add consumers when messages pile up.

HPA vs VPA vs KEDA, and how this repo actually autoscales.

recall, then click to reveal

HPA = scale POD COUNT to a metric target (usually CPU%), ~15s loop — blind to non-resource bottlenecks. VPA = resize each pod (requests/limits) from observed usage. KEDA = scale on EXTERNAL EVENTS (queue depth, Kafka lag, Prometheus) and TO/FROM ZERO; generates an HPA under the hood for 1→N. REPO: _app.tpl:39-56 — use KEDA if its CRD present, else HPA ("can't use both"). Two triggers: request-rate via Prometheus istio_requests_total (_keda.tpl:109, min 2/max 8), and Kafka lag — notification's scaledobject-consumers.yaml on sum(kafka_consumergroup_lag{...}), threshold 7500. VPA mostly updateMode: Off (recommend only).

Want to see the KEDA activation-vs-scaling phases, or why request-rate scaling reads an Istio metric? Ask me.

1. Kubernetes — Horizontal Pod Autoscaling.

2. KEDA — Scaling deployments.