# Apache Airflow Glossary

The canonical vocabulary for this workspace (Airflow 3.x). Lessons use *these* words.
Definitions are tight (what the term **is**), opinionated, and reuse other glossary terms.

## The core model

**DAG (Directed Acyclic Graph)**:
The definition of a workflow — a set of tasks plus the dependency edges that order them,
with no cycles. The unit you author (here, generated from `rules.yaml`).
_Avoid_: pipeline (informal), workflow (ok), job

**DAG Run**:
One execution of a DAG for a specific data interval. Many DAG runs of the same DAG can
exist (one per interval). A DAG is the template; a DAG run is an instance.
_Avoid_: execution, instance (ambiguous)

**Task**:
A single unit of work in a DAG — a node. Defined by an operator, a sensor, or a
`@task`-decorated function.
_Avoid_: step, job, node (ok informally)

**Task Instance**:
One run of one task within one DAG run. This is the thing that has a *state*
(running/success/failed) and gets retried.
_Avoid_: task run

**Operator**:
A pre-built, reusable task template you configure with arguments (e.g.
`SparkKubernetesOperator`, `KubernetesPodOperator`, `PythonOperator`, `BashOperator`). All
derive from `BaseOperator`.
_Avoid_: action, plugin

**Sensor**:
A special operator that *waits* for an external condition (a file, a partition, a time)
before succeeding. (⚠️ Not used in this repo — but on the certification.)
_Avoid_: poller, watcher

**TaskFlow / `@task`**:
The decorator style where a plain Python function becomes a task; Airflow wires XCom and
dependencies automatically. The modern alternative to explicit `PythonOperator`.
_Avoid_: decorator API (say TaskFlow)

**Dependency (upstream / downstream)**:
The edge that orders tasks: `a >> b` means `a` is upstream of `b` (`b` runs after `a`). You
declare tasks first, then their dependencies.
_Avoid_: link, relation, sequence

## Scheduling

**`schedule`**:
How often a DAG runs — a cron string (`"0 18 * * *"`), a preset, a timedelta, `None`
(manual), or a dataset/asset trigger. (Airflow 3 name; Airflow 2 used `schedule_interval`.)
_Avoid_: interval, frequency, cron (the value, not the field)

**`start_date`**:
The first logical date from which the DAG is scheduled. With the schedule, it defines when
the first DAG run's data interval begins.
_Avoid_: begin date, first run

**Data interval**:
The time window a DAG run covers (its `data_interval_start`/`_end`). A run for an interval
fires *after* the interval ends. The interview-critical concept behind Airflow scheduling.
_Avoid_: execution date (deprecated framing), run time

**Catchup**:
Whether the scheduler creates runs for all past intervals since `start_date` when a DAG is
enabled. `catchup=False` (our default) runs only the latest.
_Avoid_: backfill (related but distinct)

**Backfill**:
Deliberately (re)running a DAG over a range of past intervals via the CLI/command — as
opposed to catchup, which the scheduler does automatically.
_Avoid_: catchup, replay

## Execution & architecture

**Scheduler**:
The Airflow component that decides which task instances are ready to run (deps + schedule
met) and queues them. The brain.
_Avoid_: orchestrator (Airflow as a whole), cron

**Executor**:
The mechanism that actually runs queued tasks — `LocalExecutor`, `CeleryExecutor`,
`KubernetesExecutor` (this repo: Kubernetes + Local). Config, not a running process.
_Avoid_: runner, worker (the executor decides *where* workers run)

**Worker**:
The process/pod that executes a task instance. With KubernetesExecutor, each task can run
in its own pod.
_Avoid_: node, agent

**Triggerer**:
The component that runs async waits for *deferrable* operators, so a waiting task frees its
worker slot.
_Avoid_: trigger (a trigger rule is different)

**Metadata DB**:
The database (Postgres here) that stores DAG/run/task-instance state, connections,
variables — Airflow's source of truth.
_Avoid_: state store, backend

**API server / Webserver**:
The UI + REST API component. Airflow 3 splits it out as the `apiServer`; Airflow 2 called
it the `webserver`.
_Avoid_: dashboard, frontend

## Data & config

**XCom (cross-communication)**:
The small key-value channel for passing data *between tasks* (pushed by one, pulled by
another). For metadata, not big datasets.
_Avoid_: shared memory, message

**Variable**:
A global key-value setting stored in the metadata DB, read at runtime with
`Variable.get(...)` — for configuration, not per-run data.
_Avoid_: env var, param (a Param is per-DAG-run input)

**Connection**:
Stored credentials/endpoint for an external system (a DB, Slack, a K8s cluster),
referenced by a `conn_id`.
_Avoid_: secret, credential (a connection bundles them)

**Hook**:
The reusable client an operator uses to talk to an external system via a Connection. The
low-level counterpart to an operator.
_Avoid_: driver, adapter

**Trigger rule**:
The condition on a task for when it runs given its upstreams' states — default
`all_success`; others include `all_done`, `one_failed`.
_Avoid_: dependency mode

## Repo-specific

**`rules.yaml`**:
The single registry that declares every DAG (name, template, schedule, jobs); the source of
truth that `generate.py` turns into DAG `.py` files.
_Avoid_: config, manifest

**`SparkApplication`**:
The Kubernetes CRD (Kubeflow Spark Operator) that runs a Spark job; a
`SparkKubernetesOperator` task submits it.
_Avoid_: spark-submit, job

**`failed_mentions`**:
The required per-DAG list of Slack handles to alert on failure — the generator refuses to
build a DAG without it.
_Avoid_: alert list, on-call