Lesson 11 · Security, supply chain & operations
Testing at scale
The test pyramid in CI — fast unit checks on every PR, a whole cluster for E2E, and guards for generated code.
Your win: explain the levels of testing this pipeline runs — unit, drift guards, and full-stack E2E in a kind cluster — and the routing that keeps them affordable in a monorepo.
The test pyramid, CI edition
| Level | What / where | Speed |
|---|---|---|
| Unit | go test ./internal/... on every PR | seconds–minutes (many) |
| Drift guards | regenerate + git diff (DB schema, DAGs, proto) | fast (a few) |
| E2E / BDD | full stack in a real cluster | slow (few) |
The shape is the classic test pyramid: many fast unit tests at the base, few slow end-to-end tests at the top. The higher you go, the more fidelity and the more cost — so you run fewer of them, less often.
Keeping it affordable — the router (recap)
requirements router in
tiered.pre_merge.yml:82-108 runs a diff binary that
inspects the PR and decides which test jobs run. A doc-only change skips the Go tests;
a change to one service's package doesn't trigger every service's integration test
(.github/scripts/get_triggered_svcs.bash). Without this, every PR would pay for the
whole pyramid — untenable in a monorepo. And unit tests run -count=3 to catch
flakiness (Lesson 3).
E2E — a whole cluster inside the runner
deployer binary and runs
./deployments/sk.bash (:122-128) — Course 1's
kind+Skaffold local stack, in CI — waits for the services to roll out, and tears down with
sk.bash -d. So the same command you'd run locally for a production-like test is
what CI runs for E2E.
workflow_dispatch only (its push trigger is
commented out), so it's run on demand, not automatically on every merge; and (2) the
GODOG_TAGS=@critical BDD run itself is currently commented out —
the kind cluster comes up and rollouts are checked, but the Cucumber scenarios aren't executed
in this path. Knowing "what the pipeline says vs what actually runs" is exactly the
kind of precision that separates someone who's read the pipeline from someone who's operated
it.
Drift guards — testing that generated code is current
ensure-make-* (recap, in context)
A special kind of "test": ensure-make-gen-db-schema.bash and
ensure-make-generate-dags.bash regenerate the artifact and fail on
git diff (Lesson 3). Proto compatibility is guarded by buf breaking.
These don't test behaviour — they test that committed generated files match their
source, catching "someone edited the generated file / forgot to regenerate." A cheap,
high-value class of check you'll want on any codegen-heavy repo.
The test pyramid + kind in CI
Why the pyramid shape, and running a real cluster for integration tests.
→ martinfowler.com — The Practical Test Pyramid
→ kind.sigs.k8s.io — kind (used for E2E)
Check yourself (from memory)
Q1. The test pyramid says you should have…
Q2. This repo's E2E test runs the backend in…
sk.bash brings up kind+Skaffold in the runner
(dind) — Course 1's stack, in CI.
Q3. A drift guard tests that…
git diff --exit-code: committed
generated code must equal a fresh generation.
go test ./internal/...
-count=3, base) → DRIFT GUARDS (regenerate + git diff: DB schema, DAGs,
proto) → few slow E2E (top). ROUTER: the diff binary
(tiered.pre_merge.yml:82-108) decides WHICH tests run per PR — most PRs skip most
tests (monorepo affordability). E2E: tiered.post_merge_integration_test.yml brings
up the whole backend in a KIND cluster INSIDE the runner (dind) via
./deployments/sk.bash (Course 1's stack, in CI), waits on rollouts, teardown
sk.bash -d. HONEST CAVEATS: that workflow is workflow_dispatch-only
(push commented), and the GODOG_TAGS=@critical BDD run is currently COMMENTED —
the cluster comes up but scenarios don't execute in this path.get_triggered_svcs.bash selects which service tests to run, or how
the BDD run would be re-enabled? Ask me.
1. Fowler — The Practical Test Pyramid; in-repo .github/workflows/tiered.post_merge_integration_test.yml.