A standalone course · Catch up on this repo + interview prep

CDC & Streaming-ETL, from newcomer to interview-ready

How a single row change in one service's database flows — as an event — into other services and the analytics warehouse, and the control plane that keeps ~248 connectors honest.

Every service's Postgres is quietly streaming its changes. Debezium reads the write-ahead log; Kafka Connect carries each change to where it's needed; JDBC sink connectors upsert it into other services' databases and the data warehouse; ksqlDB reshapes it along the way. And because you can't hand-write hundreds of connectors, this repo generates them from declarative YAML and reconciles them with a control plane called hephaestus. This course follows one row's change end to end — grounded in the real configs and code.

How to use this Do one lesson, take its quiz from memory (no peeking), then skim the matching row of the cheat sheet. Come back a day later and re-take it — spacing beats cramming. You know the rhythm by now. Stuck? Ask me — I'm your teacher, not just the author.
One framing to hold from the start "CDC here" isn't one pipeline — it's three stacks: cross-service data-sync (replicate tables between services), the KEC warehouse, and lmsdwh (the LMS analytics warehouse, with chained CDC). Same machinery, three deployments. Lesson 1 draws the map.

Part 1 — CDC foundations & the source available now

What change-data-capture is, and how Debezium turns a row change into an event.

1 · The pipeline map

The end-to-end flow, the three stacks, and log-based vs query-based CDC.

ready

2 · Debezium & the WAL

Logical decoding (pgoutput), replication slots, and publications.

ready

3 · Snapshots

Initial vs incremental — and adding a table without recreating the connector.

ready

4 · The change event & topics

The Debezium envelope, Avro + Schema Registry, topics, and tombstones.

ready

Part 2 — Kafka Connect & the sink available now

The framework that runs it all — and the generation + control plane that make it manageable.

5 · The Kafka Connect framework

Connectors/tasks/workers/converters/SMTs, distributed mode, the REST API.

ready

6 · JDBC sink connectors

Idempotent upsert, the unwrap+route SMTs, dropped deletes.

ready

7 · The connector-generation system

Declarative YAML → hundreds of connectors, schema-driven.

ready

8 · hephaestus — the control plane

Git-as-desired-state reconcile, the delete-guard, the Helm hooks.

ready

Part 3 — Transform, warehouse & the whole pipeline available now

Reshaping the stream, landing it in the warehouse, and the end-to-end guarantees.

9 · ksqlDB — stream processing

Streams vs tables, the dim_user join, ksqlDB-owned sinks.

ready

10 · The data warehouse

The lmsdwh star schema, the medallion arc, and chained CDC.

ready

11 · The end-to-end trace

bob → auth, hop by hop; delivery semantics; the dual-write problem.

ready

12 · Operating the pipeline

Adding a table, monitoring & auto-heal — and the whole-course recap.

ready

Reference shelf

Cheat sheet

The pipeline, the source/sink configs, generation + hephaestus, interview one-liners.

Glossary

The canonical vocabulary, mapped to interview terms.

Repo CDC map

Ground truth: the three stacks, Debezium, sinks, generation, hephaestus, ksqlDB, warehouse — with file:line.

Resources

debezium.io, Confluent, ksqlDB — high-trust sources only.
All 12 lessons are built — the course is complete. You can now trace any row change across the platform, read any connector config, and explain the control plane — naming what's bespoke here (the three stacks, publications-by-migration, generation + git-as-desired-state, the delete-guard, chained CDC + tenant filters). The one thing left is a retention check — ask me to run a mock interview across the course (cold, no peeking) and I'll write the first learning records proving what stuck. Questions on any lesson? Ask me — that's what I'm here for.