Lesson 15 · Pipeline patterns & best practices
Connections, variables & secrets
Configuring a pipeline safely — creds, settings, and what never goes in code.
Your win: distinguish connections, variables, secrets, and params — and know the best practices that keep credentials out of your DAGs.
Four ways to feed config into a DAG
| Mechanism | Holds | Accessed via |
|---|---|---|
| Connection | external creds + endpoint (DB, Slack, K8s) | conn_id → a Hook |
| Variable | global config key-values | Variable.get("key") |
| Secret | sensitive values | a secrets backend (out of code) |
| Param (Lesson 12) | per-DAG-run input | params, set at trigger time |
Connections and Variables live in the metadata DB (or a secrets backend); Airflow checks a secrets backend first, then env vars, then the DB.1
Two best practices
(1) Never hard-code credentials in a DAG — use a Connection (referenced
by
conn_id) or a secrets backend. (2) Don't call
Variable.get() at the top level of a DAG file — top-level code runs
on every scheduler parse (~every 30s), so a Variable fetch there hammers the DB
(Lesson 13's "no top-level code" rule).
Anchor
Our setup: Connections come from a
connections.yaml k8s
Secret mounted into Airflow, referenced by conn_id (e.g.
slack-webhook-staging, the per-env K8s cluster connection).
Variables come from an env.json configmap
(Variable.get(...)). Secrets are SOPS-encrypted, and pods
authenticate to GCP via Workload Identity — so no long-lived keys sit in code.
Read this next
Airflow — Managing Connections + Astronomer's connections guide
How connections/variables are stored and resolved, the precedence order, and secrets backends.
→ airflow.apache.org — Managing Connections
→ astronomer.io/docs/learn — Connections
Check yourself (from memory)
Q1. An Airflow Connection stores…
Credentials + endpoint for an external system, by
conn_id, used via a Hook. Not per-run data (that's Params).
Q2. A Variable is used for…
Global key-value config (
Variable.get).
Between-task data is XCom; per-run input is a Param.
Q3. Secrets in Airflow should be…
Use a Connection / secrets backend (our SOPS + Workload
Identity). Never hard-code or log them.
Connection vs Variable vs Secret vs Param — what's each for?
recall, then click to reveal
CONNECTION — stored credentials + endpoint for an external system
(DB, Slack, K8s), by
conn_id, used via a Hook; never hard-code creds.
VARIABLE — a global key-value CONFIG setting (Variable.get("key")); config,
not per-run data. SECRET — a sensitive value kept OUT of DAG code, via a secrets backend
(our repo: SOPS-encrypted + GCP Workload Identity). PARAM — per-DAG-RUN input, set at
trigger time (Lesson 12). Best practice: don't call Variable.get at a DAG
file's top level (parsed every ~30s). Our repo mounts connections.yaml (Secret) + env.json
(Variables).
Want to see the precedence order (secrets backend → env var → UI/DB), or the
AIRFLOW_CONN_* env-var form? Ask me.