# Apache Airflow Cheat Sheet

Dense quick-reference for revision and interviews (Airflow 3.x). Terms in
[GLOSSARY.md](./GLOSSARY.md); our code specifics in
[repo-airflow-map.md](./repo-airflow-map.md).

## The model in five lines
1. A **DAG** is a workflow: **tasks** + dependency edges, no cycles.
2. A **task** is one unit of work — an **operator**, a **sensor**, or a `@task` function.
3. Each time a DAG runs you get a **DAG run** (one per **data interval**); each task in it is a **task instance** with a state.
4. The **scheduler** decides what's ready; the **executor** runs it on a **worker**.
5. State lives in the **metadata DB**; you watch it in the **UI / API server**.

## A minimal DAG (TaskFlow, Airflow 3)
```python
from airflow.sdk import dag, task          # AF3 import path
import pendulum

@dag(schedule="0 2 * * *", start_date=pendulum.datetime(2024,1,1), catchup=False)
def etl():
    @task
    def extract(): return {"n": 1}
    @task
    def load(data): ...                     # XCom wiring is automatic
    load(extract())

etl()
```
Classic-operator style: `a = PythonOperator(...); b = BashOperator(...); a >> b` (a before b).

## Operators (this repo)
| Operator | For |
|---|---|
| `SparkKubernetesOperator` | submit a `SparkApplication` CRD (60× here) |
| `KubernetesPodOperator` | run any container as a pod (python_pod / scheduling / camel) |
| `KubernetesJobOperator` | run a K8s Job (python_job) |
| `PythonOperator` | in-DAG Python (e.g. Variables → XCom) |
| `BashOperator` | small shell steps |
| sensors | ⚠️ none in this repo (but on the cert — they *wait* for a condition) |

## Dependencies
```python
a >> b          # a upstream of b (b after a)
a >> [b, c]     # fan-out
[b, c] >> d     # fan-in
```
Trigger rules control when a task runs vs its upstreams: `all_success` (default), `all_done`, `one_failed`, `none_failed_min_one_success`.

## Scheduling (the interview core)
- `schedule` = cron / preset / timedelta / `None` (manual) / dataset. (AF2: `schedule_interval`.)
- A DAG run for a **data interval** fires **after the interval ends** — not at the start.
- `start_date` + `schedule` define the first interval. Don't use a dynamic `start_date` (e.g. `now()`).
- `catchup=False` (our default): only the latest missed interval runs. `catchup=True`: all since `start_date`.
- **Backfill** = deliberately (re)run a past range via CLI; **catchup** = the scheduler doing it automatically.

## Executors
| Executor | Where tasks run |
|---|---|
| `LocalExecutor` | subprocesses on the scheduler host |
| `CeleryExecutor` | a pool of persistent worker processes |
| `KubernetesExecutor` | one pod per task instance (elastic) |
Repo runs `KubernetesExecutor,LocalExecutor`. **Deferrable** operators offload waits to the **triggerer** (free the worker slot).

## Task instance states
`none → scheduled → queued → running → success | failed | up_for_retry | upstream_failed | skipped`.
Retries: `retries` + `retry_delay` in `default_args`. Alert on failure via `on_failure_callback`.

## XCom / Variables / Connections
- **XCom** — pass small data *between tasks* (`xcom_push`/`xcom_pull`, or automatic in TaskFlow). Not for big data.
- **Variable** — global config in the metadata DB: `Variable.get("key")`.
- **Connection** — external creds/endpoint by `conn_id`; used via a **Hook**.
- **Param** — per-DAG-run input (`Param(default, type=...)`), can be set at trigger time.

## Our repo — quick facts
```
Version      Airflow 3.x (3.2.1 prod/stag, 3.1.2 local); self-hosted on GKE via official Helm chart
Executor     KubernetesExecutor,LocalExecutor; AF3 components: apiServer, scheduler, dagProcessor, triggerer, workers
DAGs         GENERATED from airflow/dags/rules.yaml by generate.py + Jinja2 templates (NOT hand-written)
Output       airflow/dags/{org}/{env}/*.py + one .yaml K8s manifest per job
Operators    SparkKubernetesOperator (dominant), KubernetesPodOperator/JobOperator, PythonOperator, BashOperator; NO sensors
Schedule     cron in rules.yaml → {{schedule}}; default "0 0 * * *"; None = manual; catchup=False everywhere
Spark        SparkApplication CRD on the Kubeflow Spark Operator (PySpark); sidecar flags: enable_xcom/sql_proxy/airflow_params
XCom         Spark writes /airflow/xcom/return.json via sidecar; Variables→PythonOperator→xcom_push→downstream
Config       connections.yaml (Secret), env.json (Variables configmap), SOPS secrets; FAB RBAC access_control in rules.yaml
Alerting     slack_alert()/slack_success() (SlackWebhookNotifier); failed_mentions REQUIRED per DAG
Deploy       generated DAGs rsynced to a GCS bucket; Airflow mounts read-only via gcsfuse (DAGS_FOLDER=/opt/airflow/dags_gcs)
Data flow    services' Postgres → Debezium CDC → Kafka Connect (hephaestus) → lmsdwh warehouse → Airflow/Spark transforms
Testing      DAG generation drift check (ensure-make-generate-dags.bash); in-pipeline Spark *-validation + Great Expectations
Note         ./local/run.bash cron ≠ Airflow (separate cron runner)
```

## Interview one-liners
- *What is Airflow?* A platform to author, schedule, and monitor workflows as code — DAGs of tasks — with retries, dependencies, and observability that a pile of cron jobs can't give you.
- *DAG vs DAG run vs task instance?* DAG = the definition; DAG run = one execution for a data interval; task instance = one run of one task within a DAG run (it has the state).
- *When does a scheduled DAG run fire?* After its data interval *ends* — e.g. a daily DAG for 2024-01-01 runs just after 2024-01-02 00:00.
- *Catchup vs backfill?* Catchup = scheduler auto-runs missed intervals on enable; backfill = you deliberately run a past range via the CLI.
- *Operator vs sensor vs TaskFlow?* Operator = pre-built task; sensor = operator that waits for a condition; TaskFlow `@task` = your Python function as a task with automatic XCom/deps.
- *Why not just cron?* Airflow adds dependencies, retries/alerting, backfills, a UI, and shared connections/variables — cron gives you none of that.
- *What's XCom for?* Passing small metadata between tasks — not moving large datasets.