Lesson 10 · This repo's Airflow

rule.yaml & dynamic DAG generation

Why you won't find a hand-written DAG in this repo.

Your win: explain how DAGs here are generated from a YAML registry, and why that beats hand-writing one .py per DAG.

DAGs are generated, not authored

Most tutorials show you a hand-written my_dag.py. This repo doesn't work that way. It has a single registry, a generator, and Jinja2 templates that produce the DAG files:

airflow/dags/rules.yaml ── registry: a `dags:` list (name, template, schedule, jobs, …) │ generate.py loads it, renders Jinja2 templates in airflow/dags/templates/ ▼ airflow/dags/{org}/{env}/{env}-{org}-{name}-{tenant}.py + one .yaml K8s manifest per job

A rules.yaml entry declares: name, which template, the schedule, tags, failed_mentions, its jobs, and per-env deployments. generate.py iterates dags → deployments → tenants → jobs and renders each into a concrete DAG + manifest.1

Why generate? With many tenants × environments × jobs, hand-writing hundreds of near-identical DAGs is error-prone. One config source (rules.yaml) + shared templates means: less duplication, and every DAG automatically gets the same alerting, access-control, retries, and structure. Change the template once, all DAGs update.
Anchor — guardrails & the "lint" The generator validates as it builds: it raises if failed_mentions is missing (generate.py:162) and if a SparkApplication name exceeds 40 chars. And CI runs a drift check (.github/scripts/ensure-make-generate-dags.bash): it regenerates and fails if the committed DAGs differ — so what's deployed always matches rules.yaml + templates. There's a second generator, generate_lmsdwh.py, for the warehouse DAGs.
How it ships make generate-dags (with AIRFLOW_ENV) writes the files; CI rsyncs them to a GCS bucket that Airflow mounts read-only via gcsfuse (DAGS_FOLDER=/opt/airflow/dags_gcs). gitSync is off. You edit rules.yaml/templates, not the generated .py.
Read this next

The airflow-job skill + Astronomer on dynamic DAGs

The repo's own how-to for adding a job, and the general pattern of generating DAGs from config.

→ .claude/skills/airflow-job/SKILL.md · repo-airflow-map.md
→ astronomer.io/docs/learn — Dynamically generating DAGs

Check yourself (from memory)

Q1. In this repo, DAGs are…

rules.yaml + generate.py + Jinja2 templates produce the .py files. You edit the config, not the output.

Q2. The main benefit of generating DAGs is…

Many tenants/envs from one config, with consistent alerting/access-control baked into the template.

Q3. The CI "lint" for DAGs checks that…

The drift check regenerates and fails if the tree changed — committed DAGs must match rules.yaml + templates.
How are DAGs authored in this repo, and why that way?
recall, then click to reveal
GENERATED, not hand-written. A single registry airflow/dags/rules.yaml lists each DAG (name, template, schedule, jobs, failed_mentions, deployments); generate.py + Jinja2 templates render the DAG .py files + per-job K8s manifests into airflow/dags/{org}/{env}/. Why: many tenants × envs × jobs → one config source beats hundreds of near-identical files and guarantees consistent alerting/access-control. The generator validates (failed_mentions required; SparkApp name ≤40), and a CI drift check (ensure-make-generate-dags.bash) fails if regenerating changes the tree. make generate-dags → GCS → gcsfuse.
Want to walk a real rules.yaml entry through to its generated .py, field by field? Ask me.

1. Repo: repo-airflow-map.md; .claude/skills/airflow-job/SKILL.md.