Lesson 9 · Scaling, scheduling & reliability
Autoscaling: HPA / VPA / KEDA
Three ways to scale — and why this repo scales consumers on Kafka lag, not CPU.
Your win: distinguish HPA (more pods), VPA (bigger pods), and KEDA (event-driven, scale-to-zero) — and explain the repo's real setup: KEDA scaling on request rate and Kafka consumer lag.
Three autoscalers, three axes
| Autoscaler | Changes | Reacts to |
|---|---|---|
| HPA (Horizontal Pod Autoscaler) | the number of Pods | a metric target, usually CPU/memory % |
| VPA (Vertical Pod Autoscaler) | each Pod's size (requests/limits) | observed usage over time |
| KEDA (Event-Driven Autoscaling) | the number of Pods, incl. to/from zero | external events: queue depth, Kafka lag, Prometheus metrics |
HPA is a control loop (Lesson 1) that adjusts a Deployment's replica count to hit a target, re-checking roughly every 15s.1 Its blind spot: it only knows resource metrics. If your bottleneck is "10,000 messages queued," CPU may look fine while you fall behind. That's what KEDA fixes.
KEDA in one line
KEDA scales a workload on external event sources — and can scale to
zero when idle, then back up when work appears. Under the hood it still generates an
HPA for the 1→N range; KEDA itself owns the 0↔1 activation.2
So it's not a competitor to HPA — it's a smarter front-end that adds event triggers and
scale-to-zero.
How this repo scales — KEDA first, HPA as fallback
Anchor — the decision logic
The library orchestrator picks the autoscaler in
deployments/helm/libs/util/templates/_app.tpl:39-56: if
the KEDA CRD (
keda.sh/v1alpha1) is installed and .Values.keda is
set, use a KEDA ScaledObject; else if .Values.hpa is set,
use a plain HPA. The code even comments "Keda is using HPA under the hood, so we can't
also use HPA" (:52). In backend, KEDA wins — plain HPA is essentially the fallback.
Meanwhile VPA is applied to nearly everything but mostly with
updateMode: Off (_vpa.tpl:45) — it
recommends sizes without auto-applying them.
The two KEDA triggers you'll see
Anchor — request-rate scaling
util.keda.requestRateScaleObject (_keda.tpl:109)
scales the main service Pods on request rate, read from a Prometheus query
over Istio's istio_requests_total metric. Defaults:
minReplicaCount 2, maxReplicaCount 8, poll every 30s, 180s cooldown
(:121-124). So traffic → more pods, quiet → fewer (down to 2).
Anchor — Kafka-lag scaling (your team's service!)
The notification team's consumers scale on Kafka consumer lag — the tie-in
to your Kafka course. backend/notificationmgmt/templates/scaledobject-consumers.yaml:4
is a
ScaledObject targeting the -consumers Deployment (Lesson 4);
its trigger queries sum(kafka_consumergroup_lag{consumergroup="…"}) (:45) and
scales up when lag crosses a threshold — 7500 in
notificationmgmt/prod-tokyo-values.yaml. This is the textbook
KEDA use case: "messages are piling up faster than we're consuming → add consumers," which
CPU-based HPA would completely miss.
Kafka topic lag climbs past 7500
│
▼
Prometheus scrapes kafka_consumergroup_lag
│
▼
KEDA ScaledObject → generates/updates an HPA → more -consumers Pods
│ (down to 0 when idle-capable)
▼
lag drops → cooldown → scale back down
Backend use case
Backend use case: this is the actual story behind KEDA-first scaling, notification consumers scaling on Kafka lag, request-rate triggers from Istio metrics, and VPA mostly in recommendation mode.
Common mistake
Common mistake: assuming CPU-based HPA explains the repo’s autoscaling behavior when KEDA and external-event triggers are doing the important work.
Read this next
Kubernetes HPA + KEDA scaling concepts
The HPA control loop and target model, then KEDA's ScaledObject, triggers, and scale-to-zero activation phase.
→ kubernetes.io — Horizontal Pod Autoscaling
→ keda.sh — Scaling deployments (ScaledObject)
In plain English
Plain English: autoscaling is about changing pod count or pod size in response to demand, and this repo often scales on event reality rather than CPU alone.
Check yourself (from memory)
Q1. HPA, VPA and KEDA respectively change…
HPA = more pods (metric), VPA = bigger pods (usage), KEDA =
more pods on external events, incl. to/from zero.
Q2. KEDA can do what a plain HPA cannot?
External-event triggers + scale-to-zero. It generates an HPA
for the 1→N range under the hood.
Q3. This repo scales notification consumers on…
A KEDA ScaledObject on
kafka_consumergroup_lag
(threshold 7500) — add consumers when messages pile up.
HPA vs VPA vs KEDA, and how this repo actually autoscales.
recall, then click to reveal
HPA = scale POD COUNT to a metric target (usually CPU%), ~15s loop —
blind to non-resource bottlenecks. VPA = resize each pod (requests/limits) from observed
usage. KEDA = scale on EXTERNAL EVENTS (queue depth, Kafka lag, Prometheus) and TO/FROM
ZERO; generates an HPA under the hood for 1→N. REPO:
_app.tpl:39-56 — use KEDA
if its CRD present, else HPA ("can't use both"). Two triggers: request-rate via Prometheus
istio_requests_total (_keda.tpl:109, min 2/max 8), and
Kafka lag — notification's scaledobject-consumers.yaml on
sum(kafka_consumergroup_lag{...}), threshold 7500. VPA mostly
updateMode: Off (recommend only).
Want to see the KEDA activation-vs-scaling phases, or why request-rate scaling reads an
Istio metric? Ask me.