Lesson 12 · This repo's Airflow

XCom, params, access-control & alerting

The rest of a job definition — passing data, runtime inputs, RBAC, and Slack.

Your win: explain how tasks pass data (XCom), take runtime params, are access-controlled, and alert — completing the picture of a job in this repo.

XCom — small data between tasks

XCom ("cross-communication") lets one task hand a small value to another (metadata, not big datasets).¹ This repo uses three flavours:

Spark → Airflow: a PySpark job writes /airflow/xcom/return.json; the enable_xcom sidecar surfaces it to Airflow (airflow/spark/src/libs/xcom.py).
Variables → XCom: a PythonOperator does Variable.get(...) then xcom_push(...), feeding a downstream task (templates/spark-job-with-airflow-variables.py).
xcom_pull into a manifest: {{ ti.xcom_pull(...) }} rendered as a later task's application_file.

Dynamic params

DAGs take runtime inputs via params (declared as Param(default, type=...)), settable at trigger time. In this repo they flow to pods as env_vars={"AIRFLOW_PARAMS": "{{ params | tojson }}"} or are mounted for Spark apps at /airflow-params/.

Access-control & alerting

Anchor — declared per DAG in rules.yaml Access-control (FAB RBAC): a DAG's access_control: block (rules.yaml:57) maps a role → permissions (can_read, can_edit, …) and renders to access_control={{access_control}} on the DAG — so who can view/trigger a DAG is config, not code. Alerting: slack_alert() / slack_success() (SlackWebhookNotifier, templates/utils/alert_utils.py) are wired as on_failure_callback / on_success_callback. failed_mentions is required — the generator refuses to build a DAG without it (Lesson 10).

Config plumbing Connections come from a connections.yaml Secret (referenced by conn_id, e.g. the Slack webhook), Variables from an env.json configmap (Variable.get(...)), and secrets are SOPS-encrypted. These are the Connections / Variables / Secrets concepts, wired the repo's way.

Airflow — XComs + Params

The core-concepts XComs page and the params/notifications docs — pair with the repo map for the wiring here.

→ core-concepts — XComs
→ core-concepts — Params

Check yourself (from memory)

Q1. XCom is for passing between tasks…

Small values by key (task_id/dag_id-scoped). For big data, pass a reference (a path), not the data.

Q2. In this repo, a DAG's allowed roles are declared in…

access_control: (FAB RBAC) per DAG → rendered onto the DAG. Config, not code.

Q3. failed_mentions is required because…

generate.py raises if it's missing — every DAG must declare who to alert on failure.

How do XCom, params, access-control, and alerting work in this repo's jobs?

recall, then click to reveal

XCOM (small metadata between tasks) three ways: a Spark job writes /airflow/xcom/return.json surfaced by the enable_xcom sidecar; a PythonOperator does Variable.get→xcom_push for a downstream task; or {{ ti.xcom_pull(...) }} renders into a later task's manifest. PARAMS: params (Param(default,type=)) settable at trigger time, passed to pods via env / mounted at /airflow-params/. ACCESS-CONTROL: FAB RBAC per DAG in rules.yaml access_control (role → permissions). ALERTING: slack_alert()/slack_success() (SlackWebhookNotifier) as on_failure_callback/on_success_callback; failed_mentions is REQUIRED (generator enforces).

🎓 That's Part 3 — this repo's Airflow You can trace the pipeline, explain the generated-DAG model, describe how Spark jobs run, and read the XCom/params/RBAC/alerting wiring. Part 4 zooms back out to the patterns that make any pipeline reliable — idempotency, sensors, config, and reliability.

Ready for the final Part 4 (pipeline patterns & best practices), or a mixed quiz across Lessons 9–12 first? Ask me.

1. Airflow — XComs; repo: repo-airflow-map.md.