Lesson 1 · CDC foundations & the source

The pipeline map

How a row change in one service's database becomes an event that reaches other services — and why this repo runs the whole thing three separate times.

Your win: draw the end-to-end pipeline, name its stages, distinguish log-based from query-based CDC, and identify the three CDC stacks so every later lesson has a place to live.

The problem CDC solves

Every service here owns its own Postgres — bob, eureka, auth, payment. But data has to cross those boundaries: auth needs a copy of bob's users; the analytics warehouse needs facts from all of them. You could have each service publish events by hand — but then a write to the DB and a publish to Kafka are two steps that can half-fail (the dual-write problem, Lesson 11). Change Data Capture avoids that: it derives the event stream from the database's own committed changes. One source of truth, no drift.1

Log-based vs query-based CDC

There are two ways to capture changes, and the choice defines everything:

Query-based (polling)Log-based (this repo)
HowSELECT … WHERE updated_at > last, on a timerread the database's write-ahead log
Deletesinvisible (the row is gone)captured (the log records them)
Missed changestwo updates between polls → you see oneevery change, in order
Loadrepeated scans on the live DB~none (reads a log the DB already writes)
Why log-based wins Reading the write-ahead log (WAL) catches every insert, update, and delete, in commit order, with almost no load on the source database — because the DB was writing that log anyway. This repo is entirely log-based, via Debezium (Lesson 2). The interview one-liner: "we read the WAL, we don't poll."

The pipeline, end to end

Every change follows the same shape. Memorize it; it's the skeleton of the whole course:

service Postgres ──WAL──▶ DEBEZIUM SOURCE ──Avro──▶ KAFKA TOPIC ──▶ [ ksqlDB? ] ──▶ JDBC SINK ──▶ destination (bob.public.users) (reads the log) local.manabie.… (transform) (upsert) (auth.users bob.public.users or dim_user) └──── source: capture ────┘ └──── Kafka ────┘ └──── transform ────┘ └──── sink: apply ────┘

Four stages: capture (Debezium reads the WAL → change event), transport (a Kafka topic per table), an optional transform (ksqlDB joins/reshapes for the warehouse), and apply (a JDBC sink upserts into the destination). Parts 1–3 walk these left to right.

The one framing to hold: three stacks

Here's the thing that confuses everyone first. "CDC here" is not one pipeline — the same machinery is deployed three times, with three separate Connect clusters, schema registries, and ksqlDB servers:

1 · Cross-service data-sync

The biggest. Debezium captures each service's Postgres; ~237 JDBC sinks replicate specific tables into other services' Postgres (bob→auth, bob→eureka). Operational data federation — no warehouse.

2 · The KEC warehouse

CDC → ksqlDB joins/aggregations → JDBC sink into an analytics warehouse.

3 · lmsdwh

The LMS analytics warehouse — a star schema, fed by chained CDC (services → lmsdwh → partner warehouses).

Say which stack Whenever a lesson says "the Connect cluster" or "the pipeline," ask which of the three. They share the exact same tech (Debezium + Connect + ksqlDB), so learning one teaches all three — but they're separate deployments. And one control plane, hephaestus, reconciles all of them (Lesson 8).
Where this lives The Debezium sources: internal/golibs/debezium/ + generated configs in local/configs/services/kafka-connect-source-connector/. The sinks: …/kafka-connect-gen-sink-connector/. The generator: internal/platform/datapipeline/. The control plane: cmd/server/hephaestus/. The repo map has every anchor.
Backend use case Backend use case: this is the framing behind why bob-to-auth sync, warehouse feeds, and chained lmsdwh CDC all use the same source→topic→sink shape.
Common mistake Common mistake: teaching “the CDC pipeline” as if this repo had one cluster instead of three separate stacks.
Read this next

What Debezium & CDC are

The clearest intro to log-based change data capture and where it fits — from the project itself, plus the mental model of "the log as the source of truth."

→ debezium.io — tutorial & overview
→ Confluent — Kafka Connect · Kleppmann — "turning the database inside out"

In plain English Plain English: CDC here means the database writes the truth once, and the pipeline turns that committed change into downstream data movement.

Check yourself (from memory)

Q1. What is the core advantage of log-based CDC over polling?

Reading the WAL sees every insert/update/delete in order, at ~no cost — polling misses deletes and changes between scans.

Q2. What are the four stages of the pipeline, in order?

Debezium captures the WAL → a Kafka topic transports it → ksqlDB optionally transforms → a JDBC sink applies it to the destination.

Q3. "The Connect cluster" in this repo refers to…

Three deployments of the same tech — cross-service sync, the KEC warehouse, and lmsdwh. Always say which one.
Recall: what CDC is, the pipeline stages, and the three stacks.
define + stages + stacks, then reveal
CDC: turn a DB's committed changes into an event stream, as they happen — one source of truth (avoids the dual-write problem). Log-based (read the Postgres WAL via Debezium) beats query-based (polling): catches every change incl. deletes, no missed updates, ~no load. Pipeline (4 stages): capture (Debezium source reads WAL → change event) → transport (a Kafka topic per table) → (transform) (ksqlDB joins/reshapes, warehouse only) → apply (JDBC sink upserts into the destination). Three stacks (separate clusters/registries/ ksqlDB): (1) cross-service data-sync (replicate tables between services, no warehouse — biggest), (2) KEC warehouse (+ksqlDB), (3) lmsdwh (star schema, chained CDC). One control plane = hephaestus.
🎯 Interview one-liner "How does data move between your microservices?" → "Log-based change data capture. Debezium reads each service's Postgres write-ahead log and emits change events to Kafka; sink connectors apply them to other services' databases and the warehouse. We read the log rather than poll, so we catch every change with no load — and the DB stays the single source of truth."
Next lesson: the capture stage up close — Debezium and the WAL: logical decoding, replication slots, and publications. Unsure which stack something belongs to? Ask me — that's what I'm here for.

1. Debezium overview, Kafka Connect. In-repo map: docs/cdc/reference/repo-cdc-map.md.