Lesson 8 · Scheduling & execution
Task lifecycle, retries & trigger rules
How a task instance moves through states — and decides whether to run at all.
Your win: name the task-instance states, explain retries, and use trigger rules to control when a task runs given its upstreams — the last piece of the execution model.
The task-instance lifecycle
A task instance (Lesson 2) is what carries state. The scheduler moves it
scheduled → queued; the executor runs it (running); it ends
success or failed.1
Retries
Set retries and retry_delay (usually in
default_args). A failed task with retries left goes up_for_retry,
waits, then runs again — automatic resilience against transient failures. Alerts fire via
on_failure_callback.
retries and max_active_runs from
rules.yaml into default_args
(common-spark-job.py:22-24), set a
dagrun_timeout of 90 minutes, and wire
on_failure_callback=[slack_alert({{failed_mentions}})] — which is why
failed_mentions is a required field (the generator refuses without it).
Trigger rules: when does a task run?
A task's trigger rule decides whether it runs based on its upstream tasks' states.2
| Rule | Runs when… |
|---|---|
all_success (default) | all upstreams succeeded (a skipped upstream cascades → skip) |
all_done | all upstreams finished, whatever their outcome (great for cleanup / notify) |
one_failed | at least one upstream failed (e.g. an error handler) |
none_failed_min_one_success | no failures and at least one success (joins after branching) |
skipped under the default
all_success, because a skipped upstream cascades. Use
none_failed_min_one_success (or all_done) so the join still
runs. This exact scenario is a favourite interview question.
Airflow — Tasks (states & trigger rules) + DAG Runs
The state diagram, retries, and the full trigger-rule list with examples.
Check yourself (from memory)
Q1. The default trigger rule is…
all_success — and a skipped upstream
cascades, which is the join-task gotcha.
Q2. A task that failed and will retry is in state…
up_for_retry while it waits
retry_delay, then runs again. upstream_failed is when a
parent failed.
Q3. trigger_rule="all_done" makes a task run…
up_for_retry and
re-runs (retries + retry_delay in default_args); a failed parent
makes it upstream_failed; an untaken branch is skipped. TRIGGER
RULES decide when a task runs given upstream states: default all_success (all
ok — and skipped cascades); all_done (any outcome — cleanup/notify);
one_failed (error handler); none_failed_min_one_success (joins
after branching). Alerts via on_failure_callback (our
slack_alert(failed_mentions)).rule.yaml becomes a DAG that runs a Spark job.
1. Airflow — Tasks.