Lesson 11 · This repo's Airflow

Spark jobs & the operators used

How a DAG task actually runs a Spark transform — via a Kubernetes CRD.

Your win: explain how a task submits a Spark job as a SparkApplication, and map each rules.yaml job type to its operator.

Airflow doesn't run Spark — it submits a CRD

A Spark task here does not spark-submit from Airflow. A SparkKubernetesOperator task creates a SparkApplication custom resource on Kubernetes; the Kubeflow Spark Operator sees the CRD and spins up the driver + executor pods, running the job and reporting status back.¹

DAG task SparkKubernetesOperator(application_file="…-{{spark_app}}-{{tenant}}.yaml") │ creates a ▼ SparkApplication CRD ──▶ Spark Operator ──▶ driver pod + executor pods (run PySpark) (rendered from deployments/helm/platforms/spark/templates/sparkapp.yaml)

Job type → operator

`rules.yaml` job type	Airflow operator
`spark_app`	`SparkKubernetesOperator` (submits a SparkApplication)
`python_job`	`KubernetesJobOperator` (+ delete-job)
`python_pod` / `scheduling` / `camel`	`KubernetesPodOperator`

The PySpark source lives in airflow/spark/src/ (shared code packaged as libs.zip); the sparkapp.yaml template renders the CRD (image, mainApplicationFile, and optional sidecars for XCom / cloud-sql-proxy / params).

Anchor In sync-learnosity-data-with-val, three of the four tasks are SparkKubernetesOperators, each with application_file="{{env}}-{{org}}-{{spark_app}}-{{tenant}}.yaml", do_xcom_push=False, and delete_on_termination=False. Counted across templates, SparkKubernetesOperator dominates (60 uses); ⚠️ no sensors are used to "wait" for the Spark job — the operator itself blocks until the SparkApplication finishes.

Kubeflow Spark Operator + the SparkKubernetesOperator

How SparkApplication CRDs work, and the Airflow provider operator that submits them.

→ kubeflow.org — Using SparkApplications
→ Airflow — cncf.kubernetes operators

Check yourself (from memory)

Q1. A Spark job in this repo runs as a…

The operator creates a SparkApplication CRD; the Spark Operator runs it as driver + executor pods.

Q2. A spark_app job maps to which operator?

spark_app → SparkKubernetesOperator; python_pod/scheduling/camel → KubernetesPodOperator.

Q3. Given a SparkApplication, the Spark Operator creates…

It launches the Spark driver + executor pods to run the PySpark code, then reports status.

How does an Airflow task run a Spark job here, and what maps to what?

recall, then click to reveal

Spark jobs run as SparkApplication CRDs on the Kubeflow Spark Operator — not spark-submit from Airflow. A SparkKubernetesOperator task submits the CRD (via application_file=<generated>.yaml, rendered from sparkapp.yaml); the Spark Operator then creates driver + executor pods and runs the PySpark (from airflow/spark/src/, shared as libs.zip), monitoring to success/failure. Job type → operator: spark_app → SparkKubernetesOperator; python_job → KubernetesJobOperator; python_pod/scheduling/camel → KubernetesPodOperator. No sensors.

Want to read a rendered SparkApplication (driver/executor cores, the libs.zip deps, the secret env)? Ask me.

1. Kubeflow — Spark Operator; Airflow cncf.kubernetes operators.