Lesson 1 · CDC foundations & the source

The pipeline map

How a row change in one service's database becomes an event that reaches other services — and why this repo runs the whole thing three separate times.

Your win: draw the end-to-end pipeline, name its stages, distinguish log-based from query-based CDC, and identify the three CDC stacks so every later lesson has a place to live.

The problem CDC solves

Every service here owns its own Postgres — bob, eureka, auth, payment. But data has to cross those boundaries: auth needs a copy of bob's users; the analytics warehouse needs facts from all of them. You could have each service publish events by hand — but then a write to the DB and a publish to Kafka are two steps that can half-fail (the dual-write problem, Lesson 11). Change Data Capture avoids that: it derives the event stream from the database's own committed changes. One source of truth, no drift.1

Log-based vs query-based CDC

There are two ways to capture changes, and the choice defines everything:

Query-based (polling)Log-based (this repo)
HowSELECT … WHERE updated_at > last, on a timerread the database's write-ahead log
Deletesinvisible (the row is gone)captured (the log records them)
Missed changestwo updates between polls → you see oneevery change, in order
Loadrepeated scans on the live DB~none (reads a log the DB already writes)
Why log-based wins Reading the write-ahead log (WAL) catches every insert, update, and delete, in commit order, with almost no load on the source database — because the DB was writing that log anyway. This repo is entirely log-based, via Debezium (Lesson 2). The interview one-liner: "we read the WAL, we don't poll."

The pipeline, end to end

Every change follows the same shape. Memorize it; it's the skeleton of the whole course:

service Postgres ──WAL──▶ DEBEZIUM SOURCE ──Avro──▶ KAFKA TOPIC ──▶ [ ksqlDB? ] ──▶ JDBC SINK ──▶ destination (bob.public.users) (reads the log) local.manabie.… (transform) (upsert) (auth.users bob.public.users or dim_user) └──── source: capture ────┘ └──── Kafka ────┘ └──── transform ────┘ └──── sink: apply ────┘

Four stages: capture (Debezium reads the WAL → change event), transport (a Kafka topic per table), an optional transform (ksqlDB joins/reshapes for the warehouse), and apply (a JDBC sink upserts into the destination). Parts 1–3 walk these left to right.

The one framing to hold: three stacks

Here's the thing that confuses everyone first. "CDC here" is not one pipeline — the same machinery is deployed three times, with three separate Connect clusters, schema registries, and ksqlDB servers:

1 · Cross-service data-sync

The biggest. Debezium captures each service's Postgres; ~237 JDBC sinks replicate specific tables into other services' Postgres (bob→auth, bob→eureka). Operational data federation — no warehouse.

2 · The KEC warehouse

CDC → ksqlDB joins/aggregations → JDBC sink into an analytics warehouse.

3 · lmsdwh

The LMS analytics warehouse — a star schema, fed by chained CDC (services → lmsdwh → partner warehouses).

Say which stack Whenever a lesson says "the Connect cluster" or "the pipeline," ask which of the three. They share the exact same tech (Debezium + Connect + ksqlDB), so learning one teaches all three — but they're separate deployments. And one control plane, hephaestus, reconciles all of them (Lesson 8).
Where this lives The Debezium sources: internal/golibs/debezium/ + generated configs in local/configs/services/kafka-connect-source-connector/. The sinks: …/kafka-connect-gen-sink-connector/. The generator: internal/platform/datapipeline/. The control plane: cmd/server/hephaestus/. The repo map has every anchor.
Read this next

What Debezium & CDC are

The clearest intro to log-based change data capture and where it fits — from the project itself, plus the mental model of "the log as the source of truth."

→ debezium.io — tutorial & overview
→ Confluent — Kafka Connect · Kleppmann — "turning the database inside out"

Check yourself (from memory)

Q1. What is the core advantage of log-based CDC over polling?

Reading the WAL sees every insert/update/delete in order, at ~no cost — polling misses deletes and changes between scans.

Q2. What are the four stages of the pipeline, in order?

Debezium captures the WAL → a Kafka topic transports it → ksqlDB optionally transforms → a JDBC sink applies it to the destination.

Q3. "The Connect cluster" in this repo refers to…

Three deployments of the same tech — cross-service sync, the KEC warehouse, and lmsdwh. Always say which one.
Recall: what CDC is, the pipeline stages, and the three stacks.
define + stages + stacks, then reveal
CDC: turn a DB's committed changes into an event stream, as they happen — one source of truth (avoids the dual-write problem). Log-based (read the Postgres WAL via Debezium) beats query-based (polling): catches every change incl. deletes, no missed updates, ~no load. Pipeline (4 stages): capture (Debezium source reads WAL → change event) → transport (a Kafka topic per table) → (transform) (ksqlDB joins/reshapes, warehouse only) → apply (JDBC sink upserts into the destination). Three stacks (separate clusters/registries/ ksqlDB): (1) cross-service data-sync (replicate tables between services, no warehouse — biggest), (2) KEC warehouse (+ksqlDB), (3) lmsdwh (star schema, chained CDC). One control plane = hephaestus.
🎯 Interview one-liner "How does data move between your microservices?" → "Log-based change data capture. Debezium reads each service's Postgres write-ahead log and emits change events to Kafka; sink connectors apply them to other services' databases and the warehouse. We read the log rather than poll, so we catch every change with no load — and the DB stays the single source of truth."
Next lesson: the capture stage up close — Debezium and the WAL: logical decoding, replication slots, and publications. Unsure which stack something belongs to? Ask me — that's what I'm here for.

1. Debezium overview, Kafka Connect. In-repo map: docs/cdc/reference/repo-cdc-map.md.