Lesson 2 · Foundations

DAGs, tasks & dependencies

The graph everything hangs off — and the trio interviews always test.

Your win: read a DAG's structure, declare dependencies, and crisply distinguish a DAG from a DAG run from a task instance.

A DAG is tasks + edges, with no cycles

A DAG (Directed Acyclic Graph) is a set of tasks connected by directed dependency edges — and it's acyclic: no task can loop back to an earlier one.¹ That "no cycles" rule is what guarantees a finite, well-defined order to run things in.

camel_job ──▶ lrn_questions ──▶ spark_app ──▶ spark_val (fetch) (transform) (build) (validate) fan-out: a ──▶ [b, c] fan-in: [b, c] ──▶ d

You declare the tasks first, then their dependencies with >>:

camel_job >> lrn_questions >> spark_app >> spark_val   # a real chain in this repo
a >> [b, c]     # fan-out: b and c both run after a
[b, c] >> d     # fan-in: d runs after both b and c

a >> b reads "a is upstream of b" — b runs after a succeeds.

The trio: DAG vs DAG run vs task instance

Memorise this distinction A DAG is the definition of the workflow. A DAG run is one execution of it for a specific data interval (many can exist — one per interval). A task instance is one run of one task within one DAG run — and it's the thing that has a state (running / success / failed) and gets retried.

Anchor The chain above is our real sync-learnosity-data-with-val DAG (airflow/dags/templates/sync-learnosity-data-with-val.py:102): a Camel fetch, then Learnosity questions, then a Spark transform, then a Spark validation — ordered by >>. ⚠️ It's generated from a template + a rules.yaml entry, not hand-written (Lesson 10) — but the DAG it produces is exactly this graph.

Airflow — DAGs, Tasks & DAG Runs

The core-concepts pages for the DAG model, tasks, and the DAG-run/data-interval idea (which we go deep on in Part 2).

→ core-concepts — DAGs · Tasks
→ DAG Runs

Check yourself (from memory)

Q1. A DAG must be acyclic, meaning…

No cycles → a finite, well-defined execution order. You can't have A depend on B that depends on A.

Q2. a >> b means…

a is upstream of b — b waits for a to succeed. Fan-out with a >> [b, c].

Q3. A single execution of a DAG for one interval is a…

DAG run = one execution for a data interval; a task instance is one task within that run.

Distinguish DAG, DAG run, and task instance; and how are dependencies declared?

recall, then click to reveal

A DAG is the DEFINITION — tasks + dependency edges, no cycles (acyclic → a finite ordering). A DAG RUN is one EXECUTION for a specific data interval (many can exist, one per interval). A TASK INSTANCE is one run of one task within one DAG run — the thing with a state (running/success/failed) and retries. Dependencies use >>: a >> b = b after a; a >> [b, c] fans out; [b, c] >> d fans in. Our sync-learnosity-data-with-val chains camel_job >> lrn_questions >> spark_app >> spark_val.

Want to see task groups, or how a diamond dependency (fan-out then fan-in) behaves on failure? Ask me.

1. Airflow — DAGs.