A course grounded in your codebase
Apache Airflow, from newcomer to interview-ready
Workflow orchestration — anchored to this repo's data-pipeline stack.
Airflow is the data-pipeline stack in this monorepo — the
airflow-job workflow, Spark jobs, the lmsdwh warehouse — sitting
downstream of the Kafka events your services emit. It's new territory for you, so
this course starts from "what is orchestration" and builds to the certification syllabus:
the DAG model, scheduling, execution, this repo's setup, and pipeline patterns. Four parts,
built one at a time. Lessons are short — one win each. Read them in order.
Part 1 — Foundations: what Airflow is & the DAG model available now
Start here: why orchestration exists, and the DAG that everything hangs off.
1 · What Airflow is & why
Workflow orchestration — and why teams reach for it over a pile of cron jobs.
ready2 · DAGs, tasks & dependencies
The directed acyclic graph — tasks and the edges that order them.
ready3 · Operators, sensors & TaskFlow
The three flavours of task — pre-built operators, waiting sensors, and @task.
4 · Airflow architecture
Scheduler, executor, workers, metadata DB, webserver — who does what.
readyPart 2 — Scheduling & execution available now
The interview-critical core — including the data-interval concept everyone gets wrong.
5 · Scheduling & data intervals
When a run actually fires (after the interval ends) and what the logical date is.
ready6 · Catchup & backfill
Auto-running missed intervals vs deliberately rerunning the past — why catchup=False.
7 · Executors
Local / Celery / Kubernetes; the executor-vs-worker mix-up; the triggerer.
ready8 · Task lifecycle & trigger rules
States, retries, and how a task decides to run given its upstreams.
readyPart 3 — This repo's Airflow available now
The concrete stack: how a rule.yaml entry becomes a DAG that runs a Spark job.
9 · The data-pipeline stack
Where Airflow fits: services → CDC → Kafka Connect → warehouse → Spark (with hephaestus).
ready10 · rule.yaml & dynamic DAGs
DAGs generated from config, not hand-written — and the CI drift check.
ready11 · Spark jobs
SparkApplication CRDs on the Spark Operator; the job-type→operator map.
12 · XCom, params, access-control & alerting
Passing data, runtime inputs, FAB RBAC, and the required Slack mentions.
readyPart 4 — Pipeline patterns & best practices available now
What separates a demo DAG from a production pipeline you trust overnight.
13 · Idempotency
The reproducible, safely-retryable task — why every rerun must be safe.
ready14 · Sensors & deferrable operators
Waiting on external things — poke vs reschedule vs deferrable.
ready15 · Connections, variables & secrets
Configuring a pipeline safely — and keeping credentials out of code.
ready16 · Reliability & testing
Retries, alerting, timeouts, data validation, and the DAG drift check.
readyPart 5 — Senior Airflow judgment new
The senior stretch — orchestration boundaries, time realism, generated DAG contracts, reliability judgment, and interview-grade answers.
17 · Airflow truthfulness
Tell the real Airflow story accurately: what the orchestrator owns here, what it does not, and why that boundary matters.
new18 · Scheduling realism
Reason clearly about intervals, logical date, catchup, backfill, and correctness.
new19 · Generated DAGs as an API
Review `rules.yaml`, templates, validations, and drift checks as a contract surface.
new20 · Reliability realism
Understand why retries and alerts are not enough, and what false-green runs can hide.
new21 · Architecture judgment
Explain executor, triggerer, Spark boundaries, and when orchestration becomes accidental complexity.
new22 · Senior interview questions
High-signal Airflow questions with answer shapes that sound like real operational experience.
new23 · Mock senior interview pack
A retrieval-first mock round with scoring and strong answer points.
newReference shelf
Cheat sheet
Dense revision sheet + interview one-liners.
Glossary
The canonical vocabulary, opinionated.
Repo Airflow map
Ground truth: rule.yaml, Spark, deployment, versions.
Resources
Official docs, Astronomer, the cert, the book, plus senior-Airflow judgment topics.
Senior backend playbook
The compressed version of the new senior part: orchestration truthfulness, scheduling realism, generated-DAG contracts, and reliability judgment.