# Mission: Apache Airflow

## Why
Walk into an Airflow interview (or the Astronomer Fundamentals certification) able to
explain *how workflow orchestration works* — DAGs, scheduling, the execution model — with a
mental model solid enough to reason about a pipeline I've never seen. Airflow is the
**data-pipeline** stack in this monorepo (the `airflow-job` workflow, Spark jobs, the
`lmsdwh` warehouse), *not* my team's gRPC services — so it's largely new to me. Grounding
every concept in this repo's real Airflow setup (how a DAG is registered from `rule.yaml`,
how a Spark job is scheduled) makes the abstractions concrete, and connects to the data my
services (conversationmgmt / notification / spike) emit upstream.

## Success looks like
- I can explain what Airflow is, why teams use it over cron, and draw its architecture
  (scheduler, executor, workers, metadata DB, webserver).
- I can read and write a DAG: tasks, operators/sensors/TaskFlow, and dependencies.
- I can reason about scheduling: `schedule`, `start_date`, data intervals, and
  catchup/backfill — and predict when a DAG run fires.
- I can name the executors (Local / Celery / Kubernetes) and the task lifecycle, retries,
  and trigger rules.
- I can trace this repo's pipeline: how a `rule.yaml` becomes a DAG that runs a Spark job,
  and how XCom / dynamic params / access-control / alerting are wired.
- I can apply pipeline best practices (idempotency, sensors, avoiding common pitfalls) and
  answer the classic interview questions without hand-waving.

## Constraints
- **Newer to Airflow** — lessons start from "what is orchestration" and build up. Assume
  general backend maturity (I've done the Go/Postgres/Kafka/gRPC courses); do NOT assume
  Airflow or data-engineering fluency.
- **Format:** interactive HTML lessons (recall quizzes) + dense Markdown references — same
  format as the [Kafka](../kafka/index.html), [Go](../golang/index.html),
  [gRPC](../grpc/index.html) & [Postgres](../postgresql/index.html) courses. Lives in
  `docs/airflow/`.
- **Multi-session:** the roadmap ([index.html](./index.html)) spans four parts; lessons are
  short and arrive part by part.
- **Four pillars in scope:** core concepts · scheduling & execution · this repo's Airflow ·
  pipeline patterns & best practices.

## Out of scope (for now)
- Deep Airflow provider/plugin authoring beyond what the repo uses.
- Airflow cluster administration/tuning (HA scheduler, DB tuning) beyond a conceptual mention.
- Non-Airflow orchestrators (Dagster, Prefect, Luigi) except as brief comparison.
- (Revisit if the mission shifts — confirm before changing.)