Lesson 7 · Scheduling & execution

Executors

Where your tasks actually run — and the executor-vs-worker mix-up.

Your win: name the executors, say what each does, and explain how deferrable operators + the triggerer make waiting cheap.

The executor decides where tasks run

The scheduler decides which task instances are ready and queues them; the executor is the configuration that decides how and where they run.¹

Executor	Runs tasks…	Good for
LocalExecutor	as subprocesses on the scheduler host	a single machine, simple setups
CeleryExecutor	on a pool of persistent worker processes	scaling out across many workers
KubernetesExecutor	one pod per task instance	elastic, isolated, containerised work

Executor ≠ worker The executor is config — the strategy. The worker is the actual process/pod that runs one task. "KubernetesExecutor" just means the strategy is "give each task instance its own pod." Getting this distinction right is a common interview checkpoint.

Anchor This repo runs a hybrid KubernetesExecutor,LocalExecutor (custom-airflow/stag-manabie-values.yaml:17). Each SparkKubernetesOperator / KubernetesPodOperator task runs as its own pod — so a heavy Spark transform gets isolated resources and can't starve the scheduler. That's exactly what KubernetesExecutor buys you.

Deferrable operators & the triggerer

A task that's just waiting (a sensor, a "wait for this job to finish") normally ties up a worker slot doing nothing. A deferrable operator instead "defers" — hands its wait to the async triggerer and releases its slot — so you can wait on thousands of things without a worker each.²

Backend use case Backend use case: the real repo posture is KubernetesExecutor plus LocalExecutor, which matters when you explain why Spark work and Airflow work are related but not the same execution layer.

Common mistake Common mistake: assuming executor choice only affects scaling, when it also changes isolation, debugging shape, and failure boundaries.

Airflow — Executor concepts (+ Astronomer's executor guide)

The trade-offs of Local / Celery / Kubernetes, and when to choose each; plus how the triggerer serves deferrable operators.

→ airflow.apache.org — Executor
→ astronomer.io/docs/learn — Executors

In plain English Plain English: the executor is the strategy for where tasks run, not the task code itself.

Check yourself (from memory)

Q1. The KubernetesExecutor runs each task…

One pod per task instance — elastic and isolated. That's how our Spark/pod tasks run.

Q2. The executor is best described as…

It's the run strategy (Local/Celery/Kubernetes). The worker is the process that actually executes.

Q3. A deferrable operator frees its worker slot by…

It hands the async wait to the triggerer and releases the slot — efficient waiting at scale.

Name the executors and what deferrable operators buy you.

recall, then click to reveal

The EXECUTOR is config for how/where queued task instances run (the scheduler queues; the executor runs). LocalExecutor — subprocesses on the scheduler host (single machine). CeleryExecutor — a pool of persistent worker processes (scale out). KubernetesExecutor — one pod per task instance (elastic, isolated). Our repo runs a hybrid KubernetesExecutor,LocalExecutor, so Spark/pod tasks each get their own pod. DEFERRABLE operators + the TRIGGERER: a waiting task can "defer" — hand its wait to the async triggerer and release its worker slot — so you can wait on thousands of things without tying up workers.

Want the CeleryKubernetes / hybrid-executor rationale, or how KubernetesExecutor uses a pod template? Ask me.

1. Airflow — Executor.

2. Airflow — Deferrable Operators & Triggers.