Lesson 2 · CDC foundations & the source

Debezium & the WAL

How Debezium reads a Postgres database's changes without touching a single table — and the three server-side objects that make it reliable.

Your win: explain how Debezium captures changes from the Postgres write-ahead log via logical decoding, and name the roles of the plugin, the replication slot, and the publication — plus the one this repo manages unusually.

The write-ahead log is the source

Postgres writes every change to its write-ahead log (WAL) before the change hits the table — that's how it survives a crash. The WAL is therefore a complete, ordered record of every insert, update, and delete. Debezium's whole trick is to read that log instead of the tables.1 But the raw WAL is a physical, binary format — so we need to turn it into logical row changes.

Logical decoding & pgoutput

Postgres's logical decoding feature converts the physical WAL into a logical stream of row changes (this row was updated, these columns changed). It needs an output plugin to format that stream. This repo uses pgoutput — the plugin built into Postgres itself, so there's nothing to install:

internal/platform/datapipeline/source.go:150,158
connector.class = "io.debezium.connector.postgresql.PostgresConnector"
plugin.name     = "pgoutput"      // native logical-decoding plugin (needs wal_level=logical)

Three server-side objects

Reliable CDC rests on three Postgres objects. Get these and you understand the source:

internal/platform/datapipeline/source.go:166-177 · local/…/kafka-connect-source-connector/bob.json:20-31
slot.name                 = "local_manabie_bob"      // {env}_{org}_{db} — the replication slot
slot.drop.on.stop         = "false"
publication.name          = "debezium_publication"
publication.autocreate.mode = "disabled"            // ← made by SQL migration, NOT Debezium
signal.data.collection    = "public.dbz_signals"      // (Lesson 3)
heartbeat.interval.ms     = "20000"                  // keeps the slot advancing

🔖 Replication slot

A named, server-side bookmark that retains WAL until Debezium has consumed it. So if Debezium goes down, Postgres holds the log — and it resumes exactly where it left off, missing nothing. Named {env}_{org}_{db}.

📢 Publication

Declares which tables logical replication carries. Debezium can create it automatically — but here autocreate.mode = disabled: publications are created and altered by SQL migrations instead.

💓 Heartbeat

On a low-traffic DB the slot can stall (the WAL position never advances, so it retains WAL forever). A periodic heartbeat write keeps the slot moving. Every 20s here.

The repo-specific bit worth knowing Publications are managed by SQL migration, not Debezium (publication.autocreate.mode = disabled). A migration runs CREATE PUBLICATION / ALTER PUBLICATION … ADD TABLE (e.g. migrations/lmsdwh/1001_migrate.up.sql:15-53, with an is_table_in_publication() guard). Why? So the set of captured tables is version-controlled and reviewed like any schema change — not silently changed by a connector config. It's the kind of "we don't let the tool do the risky thing automatically" choice that interviewers probe for.
Why you must never casually delete a source connector Deleting a Debezium source connector can drop its replication slot — and once the slot is gone, Postgres stops retaining WAL and the position is lost. Any changes written while it's gone are never captured. This is exactly why the control plane's delete logic refuses to delete source connectors (Lesson 8) — and why adding a table uses an incremental snapshot instead of recreating the connector (Lesson 3). The slot is precious.
Read this next

The Debezium PostgreSQL connector

The definitive reference on WAL/logical decoding, replication slots, publications, and the pgoutput plugin.

→ debezium.io — PostgreSQL connector
→ debezium.io — logical decoding plugins · in-repo internal/platform/datapipeline/source.go:145-198

Check yourself (from memory)

Q1. What does Debezium actually read to capture changes?

The WAL, turned into logical row changes by the pgoutput plugin — no table reads, no polling.

Q2. What is the replication slot's job?

A slot is a server-side bookmark that holds WAL until Debezium reads it — that's what makes downtime safe. (Declaring tables is the publication's job.)

Q3. How are the captured tables (the publication) managed here?

publication.autocreate.mode = disabled; a migration runs CREATE/ALTER PUBLICATION, so the captured set is version-controlled.
Recall: how Debezium captures Postgres changes + the three objects.
WAL + plugin + 3 objects, then reveal
Debezium reads the WAL (Postgres's ordered log of every change, written before the table), turned into logical row changes by the pgoutput logical-decoding plugin (native; needs wal_level=logical). Three server-side objects: replication slot ({env}_{org}_{db}, retains WAL until consumed → downtime-safe resume; slot.drop.on.stop=false); publication (debezium_publication, declares which tables — here autocreate.mode=disabled, so it's made by SQL migration, not Debezium); heartbeat (every 20s, keeps the slot advancing on quiet DBs). Never casually delete a source connector → drops the slot → WAL position lost → missed changes (hence the delete-guard + incremental snapshots). Config: source.go:145-198, bob.json.
🎯 Interview one-liner "How does Debezium read Postgres?" → "Logical decoding of the write-ahead log with the native pgoutput plugin. A replication slot retains the WAL so we resume without gaps across restarts, and a publication declares the captured tables — which we manage by SQL migration rather than letting Debezium autocreate it. Deleting a source drops the slot, so we guard against that."
Next: the WAL only carries future changes — so how do existing rows get into the stream? Snapshots, and the clever signal-table trick for adding a table without touching the slot. Ask me about slots or publications if they're fuzzy.

1. Debezium PostgreSQL connector, plugins. In-repo: internal/platform/datapipeline/source.go:145-198, local/configs/services/kafka-connect-source-connector/bob.json.