Lesson 4 · Foundations
Airflow architecture
Who does what — the components behind every DAG run.
Your win: draw Airflow's core components and say what each does — the "explain the architecture" question the Fundamentals cert opens with.
The components
DAG files (in GCS, mounted via gcsfuse)
│ parsed by
▼
DAG PROCESSOR ──▶ METADATA DB (Postgres: DAGs, runs, task instances, connections, vars)
▲
SCHEDULER ── decides which task instances are ready → queues them
│ hands to
▼
EXECUTOR (Kubernetes + Local) ── runs each task on a WORKER (a pod)
TRIGGERER ── runs async waits for deferrable operators
API SERVER ── the UI + REST API you watch it all through
| Component | Job |
|---|---|
| Scheduler | decides which task instances are ready (deps + schedule met) and queues them |
| DAG processor (AF3) | parses DAG files into the metadata DB (split out from the scheduler) |
| Executor | config for how/where queued tasks run (Local / Celery / Kubernetes) |
| Worker | the process/pod that actually runs a task instance |
| Triggerer | runs async waits for deferrable operators, freeing worker slots |
| Metadata DB | Postgres — the source of truth for all state, connections, variables |
| API server (AF3) / Webserver (AF2) | the UI + REST API |
Executor ≠ worker
A common mix-up: the executor is configuration deciding
where tasks run; the worker is the actual process/pod that runs
one. "KubernetesExecutor" means each task instance gets its own pod.
Anchor
This repo runs Airflow 3 with executor
KubernetesExecutor,LocalExecutor and the AF3 component split —
apiServer, scheduler, dagProcessor,
triggerer, workers — a bundled Postgres 16
metadata DB, and DAG files mounted from a GCS bucket via gcsfuse
(AIRFLOW__CORE__DAGS_FOLDER=/opt/airflow/dags_gcs). ⚠️ AF3 renamed the
webserver to the API server — an interview-relevant version detail.
Read this next
Airflow — Architecture Overview
The official diagram and component descriptions (scheduler, executor, workers, metadata DB, API server/DAG processor).
Check yourself (from memory)
Q1. The Airflow scheduler is responsible for…
It evaluates deps + schedule and queues ready task
instances. The executor/workers actually run them.
Q2. The executor determines…
It's config for the run target (Local/Celery/Kubernetes).
The worker is the process that executes.
Q3. Airflow's source of truth for state is the…
The metadata DB (Postgres) holds DAG/run/task-instance
state, connections, and variables.
Name Airflow's core components and what each does.
recall, then click to reveal
SCHEDULER — decides which task instances are ready (deps + schedule)
and queues them. DAG PROCESSOR (AF3) — parses DAG files into the metadata DB. EXECUTOR —
config for how/where queued tasks run (Local/Celery/Kubernetes; repo = Kubernetes+Local).
WORKERS — the processes/pods that execute task instances. TRIGGERER — runs async waits for
deferrable operators. METADATA DB (Postgres) — source of truth for all state, connections,
variables. API SERVER (AF3) / WEBSERVER (AF2) — the UI + REST API. Our repo runs all of
these on GKE with DAGs mounted from GCS via gcsfuse.
🎓 That's Part 1 — the foundations
You can say what Airflow is, read a DAG, name the task flavours, and draw the
architecture. Part 2 is the interview-critical one: scheduling
& execution — data intervals, catchup/backfill, executors, and the task
lifecycle.
Ready for Part 2 (scheduling & execution), or a mixed quiz across
Lessons 1–4 first? Ask me.