Lesson 13 · Pipeline patterns & best practices

Idempotency

The property that makes retries, backfills, and catchup safe.

Your win: explain what an idempotent task is, why Airflow requires it, and how to design one — the #1 data-pipeline best practice.

Airflow will rerun your task

Everything you learned in Part 2 reruns tasks: retries after a failure (L8), backfill over past dates (L6), catchup, and manual clears. So a task must be idempotent — running it again for the same data interval produces the same result, with no duplicate rows or side-effects.1

How to design for it Make the task a pure function of its data interval, not of now(). Write to a deterministic destination keyed / partitioned by that interval. Overwrite or UPSERT, never blindly append. Then a rerun replaces the same output instead of doubling it.
The classic anti-pattern Using datetime.today() (or now()) inside a task breaks idempotency: rerun a failed 2024-01-01 run today and it processes today's data, not Jan 1's. Use the DAG run's logical date / data interval (Lesson 5), which is stable across reruns.
Anchor Our Spark transforms are idempotent by construction: each run recomputes and overwrites the dim_*/fact_* tables for its interval (a rebuild, not an append). Combined with catchup=False (Lesson 6), a rerun is safe. This is the same instinct as Kafka's at-least-once + idempotency (Kafka L6) and Postgres UPSERT (Postgres L4) — one principle across your courses.
Read this next

Airflow Best Practices + Astronomer DAG best practices

Idempotency, avoiding top-level code, and templated fields — the foundational write-up.

→ airflow.apache.org — Best Practices
→ astronomer.io/docs/learn — DAG best practices

Check yourself (from memory)

Q1. An idempotent task is one where rerunning it…

Same result, no duplicates/side-effects — what makes retries and backfills safe.

Q2. Idempotency matters because Airflow…

Retries, backfill, catchup, and manual clears all rerun tasks — so a rerun must be safe.

Q3. An idempotent task should key its output on…

The data interval is stable across reruns; now() isn't — that's the classic bug.
What is an idempotent task, why does Airflow need it, and how do you design one?
recall, then click to reveal
An IDEMPOTENT task produces the SAME result when rerun for the same data interval — no duplicated rows or side-effects. Airflow WILL rerun tasks (retries, backfills, catchup, manual clears), so idempotency is essential for correctness. Design: make the task a pure function of its DATA INTERVAL (not now()); write to a deterministic destination keyed/partitioned by that interval; OVERWRITE or UPSERT rather than append. Our Spark transforms rebuild dim_*/fact_* tables per run — idempotent by construction. (Same instinct as Kafka at-least-once + idempotency and Postgres UPSERT.)
Want to make a non-idempotent "append to a table" task idempotent (delete-insert by partition, or MERGE)? Ask me.

1. Airflow — Best Practices.