Lesson 16 · Pipeline patterns & best practices

Reliability & testing DAGs

The practices that make a pipeline one you can trust overnight.

Your win: name the reliability practices — retries, alerting, timeouts/SLAs, data validation — and how DAGs get tested, especially the repo's way.

Surviving failure

Retries — retries + retry_delay (Lesson 8) absorb transient failures automatically.
Alerting — on_failure_callback fires when a task fails (our slack_alert(failed_mentions)).
Timeouts / SLAs — dagrun_timeout bounds a run's duration (ours = 90 min); SLA / deadline-miss callbacks warn when a task runs long.

Trusting the output — data validation

Green DAG ≠ correct data A task can "succeed" and still produce wrong data. So build validation into the pipeline: add tasks that check the output and fail the DAG if it's bad. Our repo does exactly this — dedicated *-validation Spark tasks and Great Expectations checks run as steps after the transform, so bad data stops the run and alerts.

Testing DAGs

At minimum, a DAG parse / import test catches syntax errors, cycles, and bad imports before deploy. Beyond that, unit-test the task logic (the Python/PySpark functions) separately from Airflow.¹

Anchor — the repo's approach Because DAGs here are generated (Lesson 10), the main safety net is the generation drift check (.github/scripts/ensure-make-generate-dags.bash): CI regenerates from rules.yaml + templates and fails if the committed DAGs differ — so what's deployed always matches the config, and a template mistake is caught in review. Data correctness is enforced by the in-pipeline *-validation tasks + Great Expectations rather than a PySpark pytest suite. And the generator itself validates (failed_mentions required, SparkApp name length).

The pitfalls to avoid (a recap) Non-idempotent tasks (L13), a dynamic start_date like now(), heavy top-level code (parsed every ~30s), and catchup=True with an old start_date. Getting these wrong is how "the DAG looked fine" becomes a 2am page.

Airflow Best Practices (testing) + Astronomer's testing guide

DAG-parse tests, unit-testing task logic, and the reliability checklist.

→ airflow.apache.org — Testing a DAG
→ astronomer.io/docs/learn — Testing Airflow

Check yourself (from memory)

Q1. on_failure_callback is used to…

It runs on failure — our repo wires it to slack_alert(failed_mentions).

Q2. In this repo, DAG correctness is validated mainly by…

CI regenerates and fails on any diff; data quality is checked by in-pipeline *-validation tasks + Great Expectations.

Q3. A good pipeline bounds a run's duration with…

dagrun_timeout (ours = 90 min) fails a run that overruns, so a stuck job doesn't hang forever.

Name the practices that keep an Airflow pipeline reliable.

recall, then click to reveal

(1) RETRIES + retry_delay for transient failures (L8), with alerting via on_failure_callback (our slack_alert(failed_mentions)). (2) BOUND runs with dagrun_timeout (our 90m) + SLA/miss callbacks. (3) DATA QUALITY: validate outputs as tasks (our *-validation Spark jobs + Great Expectations) so bad data fails the DAG. (4) TEST DAGs: at minimum a parse/import test (no syntax errors, no cycles) + unit-test task logic; this repo's approach is the generation DRIFT CHECK plus in-pipeline validation. (5) Design well: IDEMPOTENT tasks, catchup=False, no heavy top-level code, no dynamic start_date.

🎓 Course complete — all 16 lessons From "what is orchestration" → the DAG model → scheduling & execution → this repo's generated-DAG/Spark stack → the patterns that keep it reliable. Every concept anchored to the codebase, aligned to the Astronomer Fundamentals syllabus, and connected to your other four courses (Airflow sits at the end of the Postgres→Kafka data pipeline). The last mile is retrieval under pressure — ask me to run a mixed mock interview.

Ready for the mock interview, or want to go deeper on anything (SLAs, Great Expectations, testing generated DAGs)? Ask me.

1. Airflow — Testing a DAG; Astronomer — Testing Airflow.