Lesson 14 · Pipeline patterns & best practices

Sensors & deferrable operators

Waiting on the outside world — without wasting a worker slot.

Your win: explain what a sensor is, the poke / reschedule / deferrable modes, and when to use each — a certification topic even though this repo doesn't use them.

A sensor waits for a condition

Pipelines often must wait for something external before proceeding: a source file to land, a table partition to be ready, another DAG's task to finish. A sensor is an operator whose job is exactly that — it checks a condition on an interval and succeeds once it's met.¹ Common ones: FileSensor, ExternalTaskSensor (wait for another DAG), object-storage sensors.

Three ways to wait — cheapest last

Mode	Behaviour	Use when
poke (default)	holds a worker slot the whole time, checks periodically	short waits (poke interval < ~5 min)
reschedule	releases the slot between checks, re-queues later	long waits — frees workers
deferrable	hands the async wait to the triggerer — no slot at all	waiting at scale (Lesson 7)

The rule of thumb A naïve sensor in poke mode ties up a worker doing nothing. For anything but a very short wait, use reschedule (or a deferrable operator) so the slot is freed — otherwise a handful of long-waiting sensors can starve your whole worker pool. This trade-off is a favourite interview question.

Anchor — honest note ⚠️ This repo uses no sensors. Its DAGs chain operators directly (e.g. fetch → transform → validate), and a SparkKubernetesOperator simply blocks until its job finishes rather than a separate sensor "waiting" for it. So there's no repo example here — but the certification tests sensors, and any pipeline that depends on an external signal (a daily file, an upstream DAG) will eventually need one. Learn the concept and the poke/reschedule/deferrable trade-off.

Backend use case Backend use case: the important repo truth is that there are no sensors in real usage here, so this lesson is about conceptual literacy and design judgment more than current local practice.

Common mistake Common mistake: teaching sensors as if the repo relies on them operationally today, when the repo map explicitly says it does not.

Airflow — Sensors (+ Astronomer's sensor guide)

The sensor concept, poke vs reschedule, and how deferrable operators improve on both.

→ airflow.apache.org — Sensors
→ astronomer.io/docs/learn — Sensors

In plain English Plain English: sensors wait for something to become true, and deferrable operators free the worker slot while that waiting happens.

Check yourself (from memory)

Q1. A sensor's job is to…

It checks a condition on an interval and succeeds once met (file lands, partition ready, upstream DAG done).

Q2. A sensor in "reschedule" mode…

It releases the slot and re-queues later — best for long waits. Poke holds the slot; deferrable frees it via the triggerer.

Q3. To wait on another DAG's task, use…

ExternalTaskSensor waits for a specific task/DAG in another DAG to complete.

What is a sensor, and what are the poke / reschedule / deferrable modes?

recall, then click to reveal

A SENSOR is an operator that WAITS for an external condition before succeeding — a file landing, a partition being ready, another DAG's task finishing (ExternalTaskSensor). Modes: POKE (holds a worker slot the whole time, checking periodically — simple, for short waits); RESCHEDULE (releases the slot between checks — better for long waits); DEFERRABLE (hands the async wait to the triggerer, using no worker slot — best at scale). ⚠️ This repo uses no sensors (DAGs chain operators directly), but the cert tests them and real pipelines need them.

Want to write a deferrable sensor, or see why ExternalTaskSensor needs matching execution dates? Ask me.

1. Airflow — Sensors.