Lesson 17 · Senior Airflow judgment

Airflow truthfulness: what the orchestrator owns here, and what it does not

How to explain what Airflow really owns in this repo, what it only coordinates, and why “Airflow runs the data platform” is too vague to be operationally useful.

Your win: explain what Airflow truly owns in this repo, what Spark, Kubernetes, CDC plumbing, and warehouse code still own, and why strong senior answers are precise about those boundaries.

In plain English Plain English: Airflow is the workflow control layer here, but it is not the same thing as the full data platform it helps coordinate.

Why this page matters more than it first looks

A lot of weak Airflow explanations fail in the same way: they over-credit the orchestrator. People say “Airflow runs the pipeline” or “Airflow builds the warehouse.” Those phrases sound convenient, but they stop being useful as soon as you have to debug a failed run, review an architectural change, or explain what layer actually owns a problem.

This repo is a good place to learn the better habit because Airflow is genuinely important here. It creates DAG runs, orders tasks, surfaces run state, passes params and XCom metadata, triggers Spark-backed jobs, and drives alerting. But that still does not make Airflow the same thing as Spark execution, Kubernetes execution, CDC connector plumbing, or warehouse business logic.

That is why this page matters. It teaches a more senior kind of honesty: the ability to say what the orchestrator really owns, where that ownership stops, and why that boundary changes how you reason about failures and design choices.

The truthfulness rule A strong Airflow explanation says what the orchestrator really owns here, what neighboring systems still own, and why that boundary changes debugging and design decisions.

Start with the whole data path

The easiest mistake is to talk as if Airflow were the whole data system. It is not. Upstream services write data. CDC captures changes. Kafka Connect moves them. The warehouse stores them. Spark transforms them. Kubernetes runs the containers. Airflow coordinates and observes the workflow that ties pieces of that path together.

Once you zoom out and see the full path, Airflow becomes easier to explain accurately. It is the orchestration layer over a larger platform, not a synonym for that platform.

Backend use case In this repo, Airflow sits downstream of services’ Postgres → CDC → Kafka Connect → `lmsdwh`, then orchestrates Spark-based transforms and validation over that warehouse path rather than owning the entire path end to end.

Common mistake Saying “Airflow processes the data” when the repo’s actual processing work is performed by Spark jobs submitted as `SparkApplication` CRDs, with Airflow acting as the coordinator.

What Airflow truly owns here

Airflow owns scheduling semantics, DAG-run creation, task ordering, run/task state, orchestrator-level params and XCom behavior, callbacks and alerting, and the generated-DAG contract surface built from `rules.yaml` plus templates. Those are not small things. They are the control-plane layer of this pipeline world.

But Airflow still does not own the data transforms themselves, the executor substrate, the warehouse storage engine, or the connector plumbing upstream. Those belong to Spark, Kubernetes, Postgres, and the CDC stack.

What stronger answers sound like

The strongest answers here sound like: “Airflow is the orchestration and control-plane layer for this repo’s downstream data workflows — it schedules DAG runs, coordinates task order, surfaces state, and triggers Spark-backed jobs — but the compute, warehouse, and CDC systems still own their own responsibilities, so good debugging starts by separating orchestration failure from processing failure.”

That answer is stronger because it separates control plane from execution plane instead of merging them into one fuzzy story.

Re-read the repo map with an ownership lens

Use the repo Airflow map to separate orchestration responsibility from Spark, Kubernetes, and CDC responsibility.

→ Repo Airflow map
→ Airflow — Architecture Overview

Check yourself (from memory)

Q1. What is the most truthful short summary of Airflow’s role in this repo?

The best answer separates orchestration ownership from the neighboring systems that actually execute or store the data.

What does Airflow really own here, and what does it not?

recall, then click to reveal

It owns scheduling, DAG/run/task orchestration, control-plane state, params/XCom wiring, and alerting. It does not itself own Spark execution, Kubernetes scheduling, CDC plumbing, or warehouse storage logic.

Want a “who owns this failure?” classification drill across Airflow vs Spark vs Kubernetes? Ask me.

Sources. Repo Airflow map; official Airflow architecture docs.