Lesson 1 · Foundations

What Airflow is, and why

Workflow orchestration — and why it beats a pile of cron jobs.

Your win: explain in one breath what Airflow is and why teams reach for it over cron — grounded in the data pipeline this repo actually runs.

The problem: pipelines are more than "run this at 2am"

A real data pipeline has steps with dependencies (fetch → transform → validate → load), runs on a schedule, and fails — so it needs retries, alerts, and a way to see what happened. Plain cron gives you none of that: no dependencies, no retries, no backfill of missed days, no UI.1

Airflow's answer: workflows as code

Apache Airflow is a platform to author, schedule, and monitor workflows as code. Each workflow is a DAG — a directed acyclic graph of tasks (Lesson 2). Airflow adds, on top of "run on a schedule": task dependencies, retries & alerting, backfills over history, a UI/API for observability, and shared connections/variables.2

Cron vs Airflow, in one line Cron answers "run X at time T". Airflow answers "run this graph of steps, in order, with retries and alerts, on a schedule, and let me see and re-run it." The second is what a warehouse pipeline needs.

What this repo runs

Anchor This monorepo runs Airflow 3, self-hosted on Kubernetes (GKE). A DAG like sync-learnosity-data fetches data, runs Spark transforms to build warehouse tables, validates the result, and alerts Slack on failure — on a daily cron (airflow/dags/rules.yaml). ⚠️ Note: Airflow is the data-pipeline stack (codename hephaestus + Spark), sitting downstream of your services' data (captured via Kafka/CDC) — it's not inside conversationmgmt/notification/spike. And ./local/run.bash cron is a separate plain cron runner, not Airflow.
Cross-course tie Remember the Postgres WAL → Debezium → Kafka pipeline (Postgres L8)? Airflow sits at the end of it: your services' committed writes flow through CDC into the warehouse, and Airflow orchestrates the Spark jobs that transform them. Five courses, one data path.
Read / do this next

Airflow 101: Building Your First Workflow

The official hands-on intro — author a DAG and watch it run. Pair with the free Astronomer Academy "Airflow 101" path (aimed at the Fundamentals cert).

→ airflow.apache.org — Airflow 101
→ academy.astronomer.io — Airflow 101

Check yourself (from memory)

Q1. Airflow lets you author workflows as…

Workflows are Python DAGs — "workflows as code." (This repo generates that Python from rules.yaml, Lesson 10.)

Q2. Airflow's main advantage over cron is…

Ordered dependencies, retries/alerting, backfills, and a UI — the things cron can't do.

Q3. In this repo, Airflow orchestrates…

DAGs run Spark jobs that build the warehouse — the data pipeline, downstream of your services.
What is Airflow, and why use it over cron?
recall, then click to reveal
Apache Airflow is a platform to author, schedule, and monitor workflows as code — each workflow is a DAG (directed acyclic graph) of tasks. Over cron it adds: task DEPENDENCIES (run B only after A), RETRIES and ALERTING, BACKFILLS over past dates, a UI/API for observability, and shared CONNECTIONS/VARIABLES. In this repo it's the data-pipeline stack (Airflow 3 on GKE): DAGs like sync-learnosity-data run Spark transforms to build the warehouse, downstream of your services' data (via Kafka/CDC).
Curious how Airflow compares to Dagster/Prefect, or why "workflows as code" beats a GUI tool? Ask me. I'm your teacher for this course.

1. Airflow — Architecture Overview.

2. Airflow — Core Concepts.