# Apache Airflow Resources

Curated, high-trust sources for this workspace. Explainers in the lessons draw from here —
not from memory. Grouped **Knowledge** (learn the ideas), **Wisdom** (communities), and
**Repo** (our own code, the ground truth for this mission).

> ⚠️ Version note: Airflow's current stable is **3.x**; many teams still run **2.x**. The
> repo's version is recorded in [reference/repo-airflow-map.md](./reference/repo-airflow-map.md).
> The Astronomer certification has both an Airflow 2 and an Airflow 3 Fundamentals exam.
> Where behaviour differs by version, the lessons flag it.

## Knowledge

### Start here (official, for newcomers)
- [Airflow 101: Building Your First Workflow](https://airflow.apache.org/docs/apache-airflow/stable/tutorial/fundamentals.html)
  The official hands-on intro — build a DAG, see it run. **Use for:** the first pass on the
  whole model.
- [Astronomer Academy — Airflow 101 learning path (free)](https://academy.astronomer.io/path/airflow-101)
  A structured, free 12-module path (core concepts, the UI/CLI, writing & scheduling DAGs,
  connections, sensors). **Use for:** the guided newcomer track toward the certification.

### The authoritative reference
- [Apache Airflow — Core Concepts](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/index.html)
  The canonical source. Key pages we lean on:
  [DAGs](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html) ·
  [Tasks](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html) ·
  [Operators](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html) ·
  [Architecture Overview](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/overview.html) ·
  [DAG Runs](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dag-run.html) ·
  [TaskFlow](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/taskflow.html).
  **Use for:** the exact definition behind any claim.

### Guides & best practices
- [Astronomer — Learn (docs.astronomer.io/learn)](https://docs.astronomer.io/learn/)
  Deep, practical guides on scheduling, sensors, XComs, dynamic tasks, deferrable operators,
  and DAG best practices. **Use for:** Parts 2–4 — usually the clearest write-up per topic.
- [Airflow — Best Practices](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html)
  Official guidance: idempotency, top-level code, testing. **Use for:** the patterns part.

### Book (deep dive)
- [_Data Pipelines with Apache Airflow_, 2nd ed. — de Ruiter, Harenslak et al. (Manning)](https://www.manning.com/books/data-pipelines-with-apache-airflow-second-edition)
  The definitive book (2nd ed. covers Airflow 3: TaskFlow, dynamic task mapping,
  event-driven scheduling). **Use for:** depth after a lesson. A free copy is offered via
  [Astronomer](https://www.astronomer.io/ebooks/data-pipelines-with-apache-airflow/).

### Certification
- [Astronomer Certification — Apache Airflow Fundamentals](https://www.astronomer.io/certification/) ·
  [Airflow 3 exam](https://academy.astronomer.io/certification-exam-apache-airflow-3-fundamentals) ·
  [Study guide (PDF)](https://www.astronomer.io/uploads/Apache-Airflow-Fundamentals-Study-Guide.pdf)
  Exam = 75 MCQs, 60 min, 70% to pass; covers architecture, DAGs, task lifecycle,
  scheduling. **Use for:** aligning the course to a concrete syllabus.

### This repo's build tool (Spark/K8s)
- [Kubernetes Spark Operator — `SparkApplication`](https://www.kubeflow.org/docs/components/spark-operator/)
  The repo runs Spark jobs as `SparkApplication` CRDs; DAG tasks submit them. **Use for:**
  the Spark-integration lesson. *(Confirm the exact operator/CRD in the repo map.)*

## Wisdom (Communities)
- [Apache Airflow Slack](https://apache-airflow-slack.herokuapp.com/) — the upstream
  community. **Use for:** "is this the right pattern?" and version-specific gotchas.
- [r/apache_airflow](https://www.reddit.com/r/apache_airflow/) / [r/dataengineering](https://www.reddit.com/r/dataengineering/)
  **Use for:** real-world pipeline design and interview-experience threads.
- [Airflow GitHub Discussions](https://github.com/apache/airflow/discussions) — authoritative
  answers from committers.

## Repo (ground truth for this mission)
Our own Airflow setup — the `airflow-job` workflow, `rule.yaml` DAG generation, Spark jobs,
scheduling, XCom, and deployment — is captured in
[reference/repo-airflow-map.md](./reference/repo-airflow-map.md) (generated from a scan of
`.claude/skills/airflow-job`, `deployments/`, and the data-pipeline service). The
`airflow-job` skill is the canonical how-to for creating a job here.

## Gaps
- No single trusted Airflow interview-question bank listed — the lessons build their own
  recall quizzes from the sources above and the certification syllabus.
- If the repo runs a managed Airflow (Composer/MWAA), add that provider's docs once confirmed.