Lesson 4 · Foundations
Airflow architecture
Who does what — the components behind every DAG run.
Your win: draw Airflow's core components and say what each does — the "explain the architecture" question the Fundamentals cert opens with.
The components
DAG files (in GCS, mounted via gcsfuse)
│ parsed by
▼
DAG PROCESSOR ──▶ METADATA DB (Postgres: DAGs, runs, task instances, connections, vars)
▲
SCHEDULER ── decides which task instances are ready → queues them
│ hands to
▼
EXECUTOR (Kubernetes + Local) ── runs each task on a WORKER (a pod)
TRIGGERER ── runs async waits for deferrable operators
API SERVER ── the UI + REST API you watch it all through
| Component | Job |
|---|---|
| Scheduler | decides which task instances are ready (deps + schedule met) and queues them |
| DAG processor (AF3) | parses DAG files into the metadata DB (split out from the scheduler) |
| Executor | config for how/where queued tasks run (Local / Celery / Kubernetes) |
| Worker | the process/pod that actually runs a task instance |
| Triggerer | runs async waits for deferrable operators, freeing worker slots |
| Metadata DB | Postgres — the source of truth for all state, connections, variables |
| API server (AF3) / Webserver (AF2) | the UI + REST API |
Executor ≠ worker
A common mix-up: the executor is configuration deciding
where tasks run; the worker is the actual process/pod that runs
one. "KubernetesExecutor" means each task instance gets its own pod.
Anchor
This repo runs Airflow 3 with executor
KubernetesExecutor,LocalExecutor and the AF3 component split —
apiServer, scheduler, dagProcessor,
triggerer, workers — a bundled Postgres 16
metadata DB, and DAG files mounted from a GCS bucket via gcsfuse
(AIRFLOW__CORE__DAGS_FOLDER=/opt/airflow/dags_gcs). ⚠️ AF3 renamed the
webserver to the API server — an interview-relevant version detail.
Backend use case
Backend use case: this matters here because Airflow 3 is split into apiServer, scheduler, dagProcessor, triggerer, and workers, with KubernetesExecutor plus LocalExecutor as the real execution posture.
Common mistake
Common mistake: treating the executor like a running worker process instead of as the policy for how queued work gets executed.
Read this next
Airflow — Architecture Overview
The official diagram and component descriptions (scheduler, executor, workers, metadata DB, API server/DAG processor).
In plain English
Plain English: the scheduler decides what is ready, the executor decides where it runs, workers do the work, and the metadata DB remembers what happened.
Check yourself (from memory)
Q1. The Airflow scheduler is responsible for…
It evaluates deps + schedule and queues ready task
instances. The executor/workers actually run them.
Q2. The executor determines…
It's config for the run target (Local/Celery/Kubernetes).
The worker is the process that executes.
Q3. Airflow's source of truth for state is the…
The metadata DB (Postgres) holds DAG/run/task-instance
state, connections, and variables.
Name Airflow's core components and what each does.
recall, then click to reveal
SCHEDULER — decides which task instances are ready (deps + schedule)
and queues them. DAG PROCESSOR (AF3) — parses DAG files into the metadata DB. EXECUTOR —
config for how/where queued tasks run (Local/Celery/Kubernetes; repo = Kubernetes+Local).
WORKERS — the processes/pods that execute task instances. TRIGGERER — runs async waits for
deferrable operators. METADATA DB (Postgres) — source of truth for all state, connections,
variables. API SERVER (AF3) / WEBSERVER (AF2) — the UI + REST API. Our repo runs all of
these on GKE with DAGs mounted from GCS via gcsfuse.
🎓 That's Part 1 — the foundations
You can say what Airflow is, read a DAG, name the task flavours, and draw the
architecture. Part 2 is the interview-critical one: scheduling
& execution — data intervals, catchup/backfill, executors, and the task
lifecycle.
Ready for Part 2 (scheduling & execution), or a mixed quiz across
Lessons 1–4 first? Ask me.