Lesson 10 · Secrets, scale & operations

Values at scale

How ~106 charts × an env×org matrix stays DRY — one library, global values, and thin per-cell deltas.

Your win: explain the three mechanisms that keep this repo's Helm manageable at scale — the shared library (DRY templates), global values (DRY config), and thin per-cell overrides — and why that beats copy-pasted charts.

The scale problem, stated

~25 services × ~6 environments × several organisations = hundreds of deployable combinations, each needing a full set of K8s objects. Done naïvely that's thousands of YAML files. Three DRY mechanisms collapse it:

Duplication risk	DRY mechanism
the same objects, per service	library chart — templates written once (Lesson 6)
the same config, across services	global values — set once, read everywhere
the same cell tweaks, per env×org	layered values — thin per-cell deltas (Lesson 4)

Global values — configure once, apply everywhere

.Values.global is the one section shared across a chart and its subcharts (Lesson 4). This repo uses it as the home for anything cross-service.

Anchor — the shared global: file deployments/helm/backend/values.yaml is entirely a global: block — global.image.repository/tag, global.vpa defaults, global.hasura*, global.onDemandNodeDeployment, global.caching.redis. It's layered into every service release (Lesson 4's stack). Templates read it through helpers that fall back gracefully: libs/util/templates/_env.tpl:6-16 (util.environment) and _env.tpl:37-47 (util.vendor) try .Values.global.environment/.vendor first, then a per-chart value. So "set the image tag once" really means once, for all services.

Anchor — global as a default, per-service as an override Templates deliberately blend the two: _statefulset.tpl:12 is replicas: {{ default .Values.global.replicaCount .Values.replicaCount }}, and _app.tpl:35 does mustMergeOverwrite .Values.global.vpa (default (dict) .Values.vpa) — take the global default, let a service override it. That pattern (global fallback + local override) is how one template serves 25 services without a giant switch statement.

Thin per-cell deltas

The economy of the model Because the library supplies the objects and globals supply the shared config, a per-cell file only carries what's different. backend/bob/local-manabie-values.yaml is ~40 lines — trimmed cpu/memory, a metrics toggle — not a re-declaration of bob. Multiply that thinness across ~30 cells and the whole matrix is a manageable set of small diffs, not a wall of duplicated YAML. That is the payoff of everything in this course.

Helm docs — Subcharts & Global Values + Best Practices (values)

How global works and the conventions for structuring values so they scale.

→ helm.sh — Subcharts & Global Values
→ helm.sh — Best Practices: Values

Check yourself (from memory)

Q1. What keeps the object templates from being duplicated per service?

libs/util writes the objects once; global values DRY the config, per-cell files DRY the tweaks — three different mechanisms.

Q2. .Values.global is used for values that are…

Global is the one section shared everywhere — the repo's backend/values.yaml is entirely global (image, vpa, hasura…).

Q3. default .Values.global.replicaCount .Values.replicaCount means…

default A B = B if set, else A — so the per-service value wins, falling back to the global default.

How does this repo keep ~106 charts × env×org DRY?

recall, then click to reveal

THREE DRY MECHANISMS: (1) the LIBRARY CHART writes object templates ONCE (Lesson 6) — no per-service duplication; (2) GLOBAL VALUES (.Values.global) DRY cross-service CONFIG — repo's backend/values.yaml is entirely global: (image/tag, vpa, hasura, caching), read via util.environment/util.vendor with fallback; (3) LAYERED per-cell values carry only DELTAS (Lesson 4) — bob/local-manabie-values.yaml ≈ 40 lines. PATTERN in templates: default .Values.global.X .Values.X and mustMergeOverwrite global local = global default + service override. Net: the whole env×org matrix is small diffs, not duplicated YAML.

Want to see everything that lives in the shared global: file, or how a service overrides a global default? Ask me.

1. Helm — Subcharts & Global Values; Best Practices: Values.