Lesson 3 · Foundations

Operators, sensors & the TaskFlow API

The three flavours of task — and which this repo actually uses.

Your win: name the three ways to define a task and pick the right one — a certification staple, grounded in the operators this repo runs.

A task comes in three flavours

Operator

A pre-built, reusable task template you configure with arguments — PythonOperator, BashOperator, SparkKubernetesOperator, KubernetesPodOperator. All derive from BaseOperator.

Sensor

A special operator that waits for a condition — a file to arrive, a partition to exist, a time to pass — then succeeds.

TaskFlow @task

Your plain Python function turned into a task by the @task decorator; Airflow wires XCom and dependencies for you. The modern style.

All three produce tasks in the DAG (Lesson 2) — they're just different ways to define one.1

What this repo uses

Anchor — operator-heavy, no sensors Counted across templates: SparkKubernetesOperator (60 — submits a Spark job), KubernetesPodOperator (10 — runs a container), PythonOperator (8 — e.g. read Variables → push XCom), BashOperator (3). ⚠️ There are no sensors anywhere in airflow/ (the airflow-job skill's SparkKubernetesSensor example isn't actually used here) — and little TaskFlow, since the repo generates classic-operator DAGs. But the certification expects all three, so learn the concept even where the repo skips it.

When to use which

The rule of thumb Use an operator when a pre-built one already does the job (run a pod, a Spark app, a bash command). Use a sensor when a task's whole purpose is to wait for something external. Use TaskFlow @task for custom Python logic where you want clean code and automatic XCom.
Read this next

Airflow — Operators + the TaskFlow API

The core-concepts pages for operators (and sensors as a subclass) and the TaskFlow @task style.

→ core-concepts — Operators
→ core-concepts — TaskFlow

Check yourself (from memory)

Q1. An operator is best described as a…

A reusable template you configure with args; all derive from BaseOperator.

Q2. A sensor is an operator that…

It blocks until an external condition is met (file, time, partition). Not used in this repo, but on the cert.

Q3. The TaskFlow @task decorator turns…

Your function becomes a task, with XCom and dependencies wired automatically — the modern authoring style.
Name the three ways to define a task, and which our repo uses.
recall, then click to reveal
(1) OPERATOR — a pre-built, reusable task template configured with arguments (PythonOperator, BashOperator, SparkKubernetesOperator, KubernetesPodOperator); all derive from BaseOperator. (2) SENSOR — a special operator that WAITS for an external condition (file, partition, time). (3) TaskFlow @task — a plain Python function as a task, with XCom + dependencies wired automatically (modern style). Our repo is operator-heavy (SparkKubernetesOperator dominates), uses NO sensors, and little TaskFlow — but the certification expects all three.
Want to see the same job written as a classic operator vs a TaskFlow @task, side by side? Ask me.

1. Airflow — Tasks (operators, sensors, TaskFlow).