Lesson 14 · Pipeline patterns & best practices

Sensors & deferrable operators

Waiting on the outside world — without wasting a worker slot.

Your win: explain what a sensor is, the poke / reschedule / deferrable modes, and when to use each — a certification topic even though this repo doesn't use them.

A sensor waits for a condition

Pipelines often must wait for something external before proceeding: a source file to land, a table partition to be ready, another DAG's task to finish. A sensor is an operator whose job is exactly that — it checks a condition on an interval and succeeds once it's met.¹ Common ones: FileSensor, ExternalTaskSensor (wait for another DAG), object-storage sensors.

Three ways to wait — cheapest last

Mode	Behaviour	Use when
poke (default)	holds a worker slot the whole time, checks periodically	short waits (poke interval < ~5 min)
reschedule	releases the slot between checks, re-queues later	long waits — frees workers
deferrable	hands the async wait to the triggerer — no slot at all	waiting at scale (Lesson 7)

The rule of thumb A naïve sensor in poke mode ties up a worker doing nothing. For anything but a very short wait, use reschedule (or a deferrable operator) so the slot is freed — otherwise a handful of long-waiting sensors can starve your whole worker pool. This trade-off is a favourite interview question.

Anchor — honest note ⚠️ This repo uses no sensors. Its DAGs chain operators directly (e.g. fetch → transform → validate), and a SparkKubernetesOperator simply blocks until its job finishes rather than a separate sensor "waiting" for it. So there's no repo example here — but the certification tests sensors, and any pipeline that depends on an external signal (a daily file, an upstream DAG) will eventually need one. Learn the concept and the poke/reschedule/deferrable trade-off.

Airflow — Sensors (+ Astronomer's sensor guide)

The sensor concept, poke vs reschedule, and how deferrable operators improve on both.

→ airflow.apache.org — Sensors
→ astronomer.io/docs/learn — Sensors

Check yourself (from memory)

Q1. A sensor's job is to…

It checks a condition on an interval and succeeds once met (file lands, partition ready, upstream DAG done).

Q2. A sensor in "reschedule" mode…

It releases the slot and re-queues later — best for long waits. Poke holds the slot; deferrable frees it via the triggerer.

Q3. To wait on another DAG's task, use…

ExternalTaskSensor waits for a specific task/DAG in another DAG to complete.

What is a sensor, and what are the poke / reschedule / deferrable modes?

recall, then click to reveal

A SENSOR is an operator that WAITS for an external condition before succeeding — a file landing, a partition being ready, another DAG's task finishing (ExternalTaskSensor). Modes: POKE (holds a worker slot the whole time, checking periodically — simple, for short waits); RESCHEDULE (releases the slot between checks — better for long waits); DEFERRABLE (hands the async wait to the triggerer, using no worker slot — best at scale). ⚠️ This repo uses no sensors (DAGs chain operators directly), but the cert tests them and real pipelines need them.

Want to write a deferrable sensor, or see why ExternalTaskSensor needs matching execution dates? Ask me.

1. Airflow — Sensors.