Lesson 18 · Senior Airflow judgment

Scheduling realism: intervals, catchup, backfill, and time semantics

How to reason about schedule, logical date, intervals, catchup, and backfill like a senior engineer — especially when “when it ran” and “what data window it owns” are not the same thing.

Your win: explain why Airflow time semantics cause real production bugs, distinguish run time from data interval clearly, and reason about catchup/backfill decisions without hand-waving.

In plain English Plain English: a run’s wall-clock time and the data window it is responsible for are related, but they are not the same thing, and forgetting that causes subtle bugs.

Why this is a senior Airflow topic

Many engineers can repeat the famous Airflow gotcha: the run for a day fires after the interval ends. That is a good start. But the more senior question is what operational mistakes happen when a team only half understands that rule.

The answer is: a lot. Wrong dashboards, wrong backfills, bad rerun assumptions, confused incident investigation, and people misreading which data window a run was supposed to own. Time semantics are not a trivia topic in Airflow. They are part of the correctness model.

That is why this page matters. It is trying to move you from “I know the gotcha” to “I understand what can go wrong in production if we reason about this carelessly.”

The scheduling realism rule A strong answer explains schedule, logical date, and data interval together, then names the operational consequence of getting them wrong.

Why catchup and backfill are judgment calls

People often treat catchup and backfill as simple knobs. They are not. They express operational intent. Catchup asks whether the scheduler should create all missed intervals automatically. Backfill asks whether humans intentionally want to replay history.

That is why this repo’s `catchup=False` default matters. It is a posture decision. It says the normal operational path should avoid automatically flooding historical runs unless someone has a good reason to do that work on purpose.

Backend use case In this repo, templates set `catchup=False`, `start_date = datetime(2024, 1, 1)`, and `dagrun_timeout=90m`, which means scheduling behavior is opinionated and stable rather than left to per-DAG improvisation.

Common mistake Describing a run only by when it fired instead of also naming the data interval it was responsible for, which is often the real business meaning of that run.

What stronger answers sound like

The strongest answers here sound like: “Airflow scheduling is a time-semantics model, not just a cron string — the schedule defines intervals, logical date marks the interval start, runs fire after interval end, and catchup/backfill choices determine whether history is created automatically or intentionally replayed, which directly affects correctness and operational load.”

That answer is stronger because it moves from memorized syntax to a coherent operational mental model.

Re-read DAG runs and scheduling with a correctness lens

Use the official DAG-run docs and Astronomer scheduling guide together.

→ Airflow — DAG Runs
→ Astronomer — Scheduling in Airflow

Check yourself (from memory)

Q1. Why is scheduling realism a senior Airflow topic?

The hard part is understanding the business meaning and operational cost of time semantics, not just memorizing the API names.

What is the senior way to explain Airflow time semantics?

recall, then click to reveal

Explain schedule, data interval, and logical date as one model; say that runs fire after interval end; then name how catchup and backfill change whether history is created automatically or intentionally replayed.

Want a “what data window did this failed run really own?” drill? Ask me.

Sources. Airflow DAG-run docs; Astronomer scheduling docs; repo map.