Lesson 12 · Security, supply chain & operations

Operations & the delivery model

The levers that keep a big pipeline fast and safe — and the whole delivery model in one view.

Your win: explain the operational levers (concurrency, caching, matrix, path filters, rollback) and summarise this repo's entire delivery model — the payoff of the course.

The four levers

Lever	Does	This repo
Concurrency	group runs so a new one cancels/queues an old one	CI `cancel-in-progress: true`; deploy `false`
Caching	reuse deps/layers across runs	GCS-backed Go/lint cache; buildx GHA cache
Matrix	fan one job over combinations	deploy over orgs
Path filters	run only when relevant files change	static `paths` + the `diff` router

The concurrency detail that matters CI uses cancel-in-progress: true — push a new commit and the old PR run is cancelled (no point testing stale code). But the deploy uses cancel-in-progress: false (tbd.deploy.yml:373-375) — you must never cancel a deploy mid-rollout, or you'd leave a cluster half-updated. Same feature, opposite setting, for a good reason. A great "why?" interview moment.

Anchor — caching & filtering at monorepo scale Caching here is GCS-backed (the gcs-cache action, bucket stag-manabie-cache), not GitHub's actions/cache — a deliberate choice for large self-hosted runners. Path filtering is two-layered: static paths/ paths-ignore on triggers, plus the dynamic diff router (Lesson 3) that turns individual jobs on/off. Together they stop a monorepo from rebuilding the world on every commit — the single most important efficiency in a repo this size.

Rollback — the push-based way (recap)

Roll back = deploy an older tag No magic rollback button. Because the image tag = the release tag (Lesson 5) and deploys are push-based (Lesson 8), rolling back is re-deploying a previous release tag via workflow_dispatch. Git stays the source of truth; the "undo" is auditable (a deploy of a known tag), not a mystery in-cluster edit.

The whole delivery model, in one paragraph

The payoff of the course A PR gets the Ok to test label → the diff router picks the checks → lint/unit/build/proto/drift/gitleaks gate the merge (Lesson 3). Merge to develop cuts a release tag; a 2-hourly cron builds it with docker buildx (tag = release tag, skip-if-exists), scans it, and signs a binauthz attestation (audit-only) (L5, L10). It auto-deploys to staging; prod is a manual dispatch behind a privileges-check (L6). Deploy = skaffold deploy → helm upgrade, matrixed per org, waiting for a healthy rollout (L8). It all runs on self-hosted ARC runners (K8s pods, dind for build/E2E, dynamic sizing, L7), authenticating to GCP keylessly via OIDC/WIF with per-purpose bots (L9). It's a push-based system — simple, explicit, with drift as the trade-off. Being able to narrate that, and name what's real vs aspirational (binauthz DRYRUN, BDD commented), is the mark of someone who understands their delivery pipeline.

Backend use case Backend use case: this is the synthesis lens for concurrency groups, GCS cache, org matrix fan-out, diff-based gating, redeploy-by-tag rollback, and the full delivery model recap.

Common mistake Common mistake: remembering each CI/CD feature separately instead of remembering the whole delivery system as one operational model.

GitHub — Concurrency, caching & security hardening

The concurrency/caching syntax and the hardening guide for a production pipeline.

→ docs.github.com — Caching
→ Security hardening for GitHub Actions

In plain English Plain English: operating CI/CD well means understanding the levers that trade speed, safety, cost, and deploy predictability against each other.

Check yourself (from memory)

Q1. Why does the deploy use cancel-in-progress: false?

Cancelling a rollout leaves a half-updated cluster. CI cancels stale runs (true); deploys must not.

Q2. This repo avoids rebuilding everything per commit via…

Static paths + the dynamic router turn off jobs whose files didn't change — essential in a monorepo.

Q3. Rolling back a bad deploy means…

Push-based: dispatch a deploy of an older tag (image tag = release tag). Auditable, git-sourced.

The ops levers + the whole delivery model in one breath.

recall, then click to reveal

LEVERS: CONCURRENCY (CI cancel-in-progress: true — drop stale runs; deploy false — NEVER cancel a rollout); CACHING (GCS-backed Go/lint + buildx GHA cache); MATRIX (deploy over orgs); PATH FILTERS (static paths + the diff router → don't rebuild the monorepo per commit). ROLLBACK = redeploy an OLDER release tag via dispatch (image tag = release tag; push-based, auditable). WHOLE MODEL: PR (Ok to test) → diff router → lint/unit/build/drift gate merge → tag → 2h cron → buildx build (skip-if-exists) + scan + binauthz sign (DRYRUN) → auto-staging / manual-prod (privileges-check) → skaffold deploy=helm upgrade per org, wait healthy → all on ARC runners (dind) + keyless OIDC/WIF. PUSH-BASED (drift = trade-off).

🎓 Course complete — all 12 lessons From "what CI/CD is" through the build, the trunk-based deploy chain, self-hosted runners, and the security/supply-chain layer, you can now narrate exactly how this repo turns a push into a running deployment — and, the rarer skill, name what's real vs aspirational (binauthz DRYRUN, BDD commented, no GitOps). The next step is the senior extension. You now have the delivery foundation; next comes deploy truthfulness, partial guardrails, rollback judgment, supply-chain realism, and CI/CD design review. After that, Course 6 (Observability) is still next: the dashboards this pipeline and the mesh feed.

Ready for Course 6 (Observability)? Or — eleven courses' worth of lessons, still no retention check — let me run a mock interview on CI/CD and record where you're solid. Ask me.

1. GitHub — Security hardening; Caching.