Lesson 9 · Secrets, scale & operations

SOPS secrets in Helm

Encrypted secrets in git — and this repo's twist: it doesn't decrypt them at deploy time at all.

Your win: explain the problem SOPS solves, the standard helm-secrets flow, and this repo's non-standard approach — ship the ciphertext into a Secret via .AsSecrets and decrypt in-app. Getting this right is the interview-differentiator.

The problem: secrets in a GitOps world

Config lives in git (Lesson 4). But you can't commit plaintext passwords. SOPS (Secrets OPerationS) encrypts the values of a YAML file — keys stay readable, values become ciphertext — backed by a KMS/PGP key.¹ So the encrypted file is safe to commit; only holders of the key can decrypt.

The standard flow (what most teams do) The helm-secrets plugin wraps Helm: at deploy it calls SOPS to decrypt your encrypted values file, feeds the plaintext to Helm as normal values, and Helm renders a Secret from them.² Plaintext exists briefly at deploy time, in the CI runner. That's the flow most interview answers describe — and it's not quite what this repo does.

This repo's twist — decrypt in the app, not at deploy

Anchor — ciphertext goes straight into a Secret Here the encrypted files are never decrypted at deploy time. The chart packs them as-is into a Kubernetes Secret using a Helm built-in: libs/util/templates/_secret.tpl:31-33 globs secrets/<vendor>/<env>/*.encrypted.yaml and calls ($.Files.Glob $glob).AsSecrets — .AsSecrets base64-packs the matching files into the Secret's data. The encrypted bytes are what lands in the cluster.

Anchor — the Go app decrypts at runtime The Secret is mounted into the Pod, and the server is started with --secretsPath=…secrets.encrypted.yaml (Course 2 L6; also the migrate hook, _hook_migration.tpl:49). The app itself calls SOPS in-process to decrypt, authenticating to GCP KMS via Workload Identity (the ServiceAccount annotation from Course 2 L12). So plaintext exists only inside the running process — never in git, never in etcd, never on a CI runner.

Say this precisely in an interview The helm-secrets plugin is installed by tooling (deployments/tools.bash:207-219, sops v3.7.3 + the plugin) but is not wired into the Skaffold deploy path — grep the skaffold files and there's no secrets:// reference. If you say "we use helm-secrets to decrypt values at deploy," you'd be describing the common pattern, not this repo. The accurate line: "we ship SOPS ciphertext into a Secret via .AsSecrets and decrypt in-app with Workload Identity."

A neat detail — prod fallback merge For dorp/trial cells, _secret.tpl:35-38 loads the prod secrets too and mustMergeOverwrites the cell's values on top — so a DR/trial environment inherits prod secrets unless it overrides them. A tidy use of the Lesson 3 merge functions on encrypted data.

SOPS + helm-secrets (know the standard, then the repo's variant)

How SOPS encrypts values, and how the helm-secrets plugin normally decrypts at deploy — so you can articulate exactly how this repo differs.

→ github.com/getsops/sops
→ github.com/jkroepke/helm-secrets

Check yourself (from memory)

Q1. SOPS encrypts a YAML file by…

Keys stay readable, values become ciphertext (KMS/PGP-backed) — so the file is safe to commit and diffs stay meaningful.

Q2. In this repo, secrets are decrypted…

The ciphertext is packed into a Secret (.AsSecrets); the Go app decrypts in-process using Workload Identity → KMS.

Q3. .AsSecrets (a Helm built-in) is used to…

It base64-packs files from .Files.Glob into a Secret. It does not decrypt — the encrypted bytes go into the cluster.

SOPS + how THIS repo handles secrets (vs the standard helm-secrets flow).

recall, then click to reveal

SOPS encrypts the VALUES of a YAML (keys readable, values ciphertext, KMS/PGP-backed) → safe to commit. STANDARD FLOW: the helm-secrets plugin decrypts the values file at DEPLOY time, feeds plaintext to Helm, Helm renders a Secret. THIS REPO'S TWIST: it does NOT decrypt at deploy — _secret.tpl:31-33 globs secrets/<vendor>/<env>/*.encrypted.yaml and packs the ciphertext into a K8s Secret via the Helm built-in .AsSecrets; the Go APP decrypts IN-PROCESS at runtime (--secretsPath) using GCP KMS via Workload Identity. helm-secrets IS installed (tools.bash) but NOT wired into Skaffold. dorp/trial merge prod secrets (_secret.tpl:35-38). Plaintext lives only in the running process.

Want to see how the app authenticates to KMS (Workload Identity, Course 2 L12), or why committing ciphertext keeps git diffs useful? Ask me.

1. SOPS — Secrets OPerationS.

2. helm-secrets plugin (the standard deploy-time decrypt flow this repo does not use).