Lesson 7 · Scheduling & execution

Executors

Where your tasks actually run — and the executor-vs-worker mix-up.

Your win: name the executors, say what each does, and explain how deferrable operators + the triggerer make waiting cheap.

The executor decides where tasks run

The scheduler decides which task instances are ready and queues them; the executor is the configuration that decides how and where they run.¹

Executor	Runs tasks…	Good for
LocalExecutor	as subprocesses on the scheduler host	a single machine, simple setups
CeleryExecutor	on a pool of persistent worker processes	scaling out across many workers
KubernetesExecutor	one pod per task instance	elastic, isolated, containerised work

Executor ≠ worker The executor is config — the strategy. The worker is the actual process/pod that runs one task. "KubernetesExecutor" just means the strategy is "give each task instance its own pod." Getting this distinction right is a common interview checkpoint.

Anchor This repo runs a hybrid KubernetesExecutor,LocalExecutor (custom-airflow/stag-manabie-values.yaml:17). Each SparkKubernetesOperator / KubernetesPodOperator task runs as its own pod — so a heavy Spark transform gets isolated resources and can't starve the scheduler. That's exactly what KubernetesExecutor buys you.

Deferrable operators & the triggerer

A task that's just waiting (a sensor, a "wait for this job to finish") normally ties up a worker slot doing nothing. A deferrable operator instead "defers" — hands its wait to the async triggerer and releases its slot — so you can wait on thousands of things without a worker each.²

Airflow — Executor concepts (+ Astronomer's executor guide)

The trade-offs of Local / Celery / Kubernetes, and when to choose each; plus how the triggerer serves deferrable operators.

→ airflow.apache.org — Executor
→ astronomer.io/docs/learn — Executors

Check yourself (from memory)

Q1. The KubernetesExecutor runs each task…

One pod per task instance — elastic and isolated. That's how our Spark/pod tasks run.

Q2. The executor is best described as…

It's the run strategy (Local/Celery/Kubernetes). The worker is the process that actually executes.

Q3. A deferrable operator frees its worker slot by…

It hands the async wait to the triggerer and releases the slot — efficient waiting at scale.

Name the executors and what deferrable operators buy you.

recall, then click to reveal

The EXECUTOR is config for how/where queued task instances run (the scheduler queues; the executor runs). LocalExecutor — subprocesses on the scheduler host (single machine). CeleryExecutor — a pool of persistent worker processes (scale out). KubernetesExecutor — one pod per task instance (elastic, isolated). Our repo runs a hybrid KubernetesExecutor,LocalExecutor, so Spark/pod tasks each get their own pod. DEFERRABLE operators + the TRIGGERER: a waiting task can "defer" — hand its wait to the async triggerer and release its worker slot — so you can wait on thousands of things without tying up workers.

Want the CeleryKubernetes / hybrid-executor rationale, or how KubernetesExecutor uses a pod template? Ask me.

1. Airflow — Executor.

2. Airflow — Deferrable Operators & Triggers.