Where Airflow sits — the end of a path that starts in your own services.
Your win: place Airflow in this repo's data flow, so the next three
lessons (rule.yaml, Spark, XCom) have a home — and see how it connects to the Kafka and
Postgres courses you've already done.
Airflow is the top of a pipeline, not an island
Airflow orchestrates the last stage of a longer path. Data originates in your
app services and flows through change-data-capture into a warehouse, where Airflow runs the
Spark jobs that shape it:
The cross-course payoff
This is the Postgres WAL → Debezium → Kafka pipeline you learned in the
Postgres course (Lesson 8) — with Airflow at the end. A committed write in your
service's Postgres becomes a Kafka CDC event, lands in the warehouse, and an
Airflow-scheduled Spark job transforms it. Five courses, one continuous data path.
Anchor — hephaestus vs your serviceshephaestus (internal/hephaestus) is the
CDC/connector plumbing — a set of bootstrap jobs that set up the publication
(migrations/lmsdwh/1001), upsert Kafka connectors, and
migrate the warehouse. It is not a gRPC app. Your three services
(conversationmgmt/notification/spike) are decoupled upstream sources:
they have no Airflow dependency — the pipeline reads their data downstream via Kafka/CDC.
Read this next
Repo map + change-data-capture
The full flow with file:line refs is in the repo map; pair with the Kafka course's
Debezium/CDC tie-in for the "how data gets here" half.
It orchestrates the final Spark transforms over the
warehouse — downstream of CDC and Kafka.
Q2. Data reaches the warehouse from the app services via…
Logical-replication CDC captured by Debezium, streamed by
Kafka Connect (connectors managed by hephaestus).
Q3. hephaestus is…
Bootstrap jobs that build the CDC → warehouse plumbing;
Airflow/Spark run on top of it.
Trace the data path that Airflow sits at the end of, in this repo.
recall, then click to reveal
App services' Postgres (conversationmgmt/notification/spike) →
Debezium / logical replication captures changes (CDC; publication set up in
migrations/lmsdwh/1001) → Kafka Connect streams them (hephaestus upserts the connectors)
→ the lmsdwh warehouse (Postgres) → Airflow-orchestrated Spark transforms build
dim_*/fact_* tables. hephaestus is the CDC/connector plumbing (bootstrap jobs, not a gRPC
app); the three app services are decoupled upstream sources with no Airflow dependency.
It's the Postgres-WAL→Kafka pipeline with Airflow at the end.
Want to see a real Spark transform (a dim_students / fact_*
build) or how a connector maps a source table to a warehouse table? Ask me.
Ground truth:repo-airflow-map.md — the data-pipeline flow with file:line references.