Lesson 3 · Ship it — code → prod plumbing

Secrets / SOPS / KMS

How credentials live safely in git and reach a running service without plaintext ever touching disk — plus the separate second layer that guards secrets already inside the database.

Your win: explain envelope encryption with SOPS + KMS, and describe how a Go service here gets its decrypted secrets at boot.

SOPS encrypts values; KMS holds the key

Secrets are committed to git — encrypted — using SOPS. The trick is envelope encryption:

each secret VALUE ──encrypted with──▶ a random data key (DEK), AES-256-GCM the DEK ──encrypted with──▶ GCP KMS master key (stored in the file's `sops:` block) to decrypt: ask KMS to unwrap the DEK, then use the DEK to decrypt the values

So a committed *.secrets.encrypted.yaml looks like normal YAML with the keys readable and each value replaced by ENC[AES256_GCM,data:…,iv:…,tag:…], plus a trailing sops: block holding the KMS-wrapped data key. You never need the data key itself — only access to the KMS master key — which means access is centrally granted and revocable.1

The rules file is generated Which KMS key encrypts which file is decided by .sops.yaml — 317 rules mapping a file path_regex to a gcp_kms key (per service/env/org, e.g. cryptoKeys/prod-eureka). It's generated by a script and marked "DO NOT EDIT" — a wrong regex silently picks the wrong key, so you regenerate it, never hand-edit.1

The flagship: services decrypt in-process, at boot

Here's the part people get wrong. You might expect an init-container to decrypt secrets to a file that the app then reads. For the Go services, that's not what happens — the service is handed the encrypted file path and decrypts it itself, in memory:

internal/golibs/configs/load.go:12,19,109
import "github.com/getsops/sops/v3/decrypt"

func LoadAll[T any](commonPath, configPath, secretPath string) (*T, error) {
    return loadAll[T](commonPath, configPath, secretPath, decrypt.File)   // SOPS decrypt fn
}
// DecryptFile: SOPS-decrypts secretPath in memory, YAML-merges it OVER config (highest precedence)
Why in-process decryption matters The plaintext secret never lands on disk — it exists only in the process's memory. The container mounts the encrypted file read-only; at boot configs.MustLoadAll calls SOPS, which reaches GCP KMS (via GOOGLE_APPLICATION_CREDENTIALS) to unwrap the data key, decrypts the values, and merges them over the plaintext config — with the secret taking highest precedence. So the layering is: common config < service config < decrypted secrets.2

The one exception: platform components (unleash, kafka-connect, NATS) do use a mozilla/sops sidecar that writes a decrypted file into a shared volume — but the Go backend services never do.

The second layer: AES for secrets already in the DB

SOPS protects secrets at rest in git. A separate, independent layer protects secrets at rest in Postgres: some private keys are AES-encrypted by the application before they're stored, and decrypted on read.

internal/usermgmt/…/repository/domain_api_keypair.go:88
encrypted, _ := crypt.AESEncrypt(privateKey, r.EncryptedKey, r.InitialVector)   // before INSERT
// ... on read (:127,:170): crypt.AESDecryptBase64(...) to recover the private key

This is the OpenAPI keypair from your HMAC-auth work: the API private key sits in the api_keypair table AES-encrypted, keyed by material from config (which is itself a SOPS secret). Two layers, two jobs: SOPS keeps secrets out of git; app-level AES keeps DB-resident secrets unreadable even to someone with a database dump.3

Read this next

SOPS — the docs

How SOPS encrypts values and integrates with GCP KMS — match it to a *.encrypted.yaml in the repo.

→ getsops.io/docs
→ in-repo .sops.yaml, internal/golibs/configs/load.go, internal/golibs/crypt/

Check yourself (from memory)

Q1. In SOPS + KMS envelope encryption, what does KMS actually do?

Values are AES-encrypted with a random data key; KMS encrypts (wraps) that data key. You need KMS access to unwrap it — not the key itself.

Q2. How does a Go service here get its decrypted secrets?

configs.LoadAll calls SOPS decrypt.File at boot — plaintext lives only in memory, never on disk. (Sidecars are only for unleash/kafka/nats.)

Q3. Why AES-encrypt a private key that's already in the database?

A second layer: SOPS keeps secrets out of git; app-level AES keeps DB-resident secrets unreadable to anyone holding just a DB dump.
Recall: SOPS + KMS envelope encryption, in-process decrypt, the second layer.
envelope + rules file + boot decrypt + AES, then reveal
SOPS: encrypts each value (AES-256-GCM), keys stay readable; the random data key (DEK) is wrapped by GCP KMS and stored in the sops: block (envelope encryption — you need KMS access, not the DEK). .sops.yaml (317 rules, path_regex → gcp_kms/prod-{svc}) is generated, do not edit. In-process decrypt (flagship): Go services are handed the encrypted file and call SOPS decrypt.File at boot (configs.LoadAll, load.go:12,19,109) — plaintext only in memory, secret = highest config precedence. Sidecar decrypt only for unleash/kafka/nats. Second layer: DB-stored private keys AES-encrypted by the app (golibs/crypt; OpenAPI keypair) so a DB dump alone reveals nothing.
🎯 Interview one-liner "How do you manage secrets?" → "SOPS encrypts the values in our config files with envelope encryption — a data key wrapped by GCP KMS — so we commit encrypted secrets to git. Services decrypt in-process at boot, so plaintext never hits disk. Secrets that live in the database, like private keys, get a second app-level AES layer, keyed from config, so a database dump alone reveals nothing."
Last in Part 1: the build system — how all of this becomes one binary in one image that runs as any service. Ask me if envelope encryption still feels circular (it's the standard pattern).

1. SOPS docs; in-repo .sops.yaml:1 (generated, DO NOT EDIT; 317 rules).

2. In-repo (verified): internal/golibs/configs/load.go:12,19,109-117 (decrypt.File, in-process, highest precedence); local/docker-compose.decrypt.yaml (sidecar exception).

3. In-repo: internal/golibs/crypt/{aes_gcm.go,aes.go}; internal/usermgmt/modules/user/adapter/postgres/repository/domain_api_keypair.go:88,127,170.