Lesson 6 · Scheduling & execution

Catchup & backfill

Running (or deliberately not running) missed intervals.

Your win: distinguish catchup from backfill, and explain why catchup=False is almost always the right default — a favourite "what went wrong?" interview scenario.

Catchup: the scheduler fills in the past

When you enable a DAG whose start_date is in the past, what happens to all the intervals between then and now? That's catchup. With catchup=True, the scheduler creates a DAG run for every missed interval. With catchup=False, it runs only the latest one.1

The classic disaster A start_date two years ago + catchup=True + a daily schedule = ~730 DAG runs kick off the instant you enable the DAG, hammering your warehouse. This is why catchup=False is the safe default — turn it on only when you genuinely want to process history.
Anchor Every template in this repo sets catchup=False (airflow/dags/templates/common-spark-job.py:24) — and it must, because start_date is hard-coded to datetime(2024, 1, 1). With catchup=True, enabling any DAG would trigger years of backfilled runs. No template opts into catchup.

Backfill: you run the past on purpose

Backfill is the flip side: you deliberately (re)run a DAG over a specific past date range, via the CLI or API — for reprocessing after a bug fix, or running a brand-new pipeline over historical data.2

Catchup vs backfill in one line Catchup is the scheduler running missed intervals automatically when a DAG is enabled. Backfill is you running a past range on demand. Same "process old intervals" outcome; different trigger.
Read this next

Airflow — DAG Runs (catchup) + Backfill

The official catchup behaviour and the (Airflow 3) dedicated backfill page.

→ core-concepts — DAG Runs / catchup
→ core-concepts — Backfill

Check yourself (from memory)

Q1. catchup=True makes the scheduler…

It creates a run for every interval since start_date — which can be hundreds at once. catchup=False runs only the latest.

Q2. Our DAGs set catchup=False to…

With start_date=2024-01-01, catchup=True would backfill years of runs on enable. False = only the latest.

Q3. Backfill is used to…

On-demand reprocessing of a chosen date range (CLI/API) — e.g. after a bug fix. Catchup is the automatic-on-enable version.
Catchup vs backfill — define each and say why catchup=False is the usual choice.
recall, then click to reveal
CATCHUP: the scheduler AUTOMATICALLY creating runs for every interval missed since start_date when a DAG is enabled. With catchup=True and a past start_date, enabling the DAG triggers a flood of historical runs — so catchup=False (our default everywhere) runs only the latest, which you almost always want. BACKFILL: YOU deliberately (re)running a past date range via CLI/API — for reprocessing after a bug fix or a new pipeline over old data. Same "run past intervals" outcome; catchup is automatic-on-enable, backfill is on-demand. Our templates hard-code start_date=2024-01-01, so catchup=True would be catastrophic.
Want to know how to safely backfill just one bad day, or why a dynamic start_date (e.g. now()) is a bug? Ask me.

1. Airflow — DAG Runs (Catchup).

2. Airflow — Backfill.