Lesson 3 · CDC foundations & the source

Snapshots

The WAL only holds what happens next — so how do the rows that already exist get into the stream? And how do you add a table without losing the slot?

Your win: explain why snapshots exist, distinguish initial from incremental snapshots, and describe the signal-table trick this repo uses to add a table to CDC without recreating the source connector.

The gap the WAL can't fill

A replication slot only streams changes from the moment it's created forward. But a table already has millions of existing rows — and the destination needs those too, not just future edits. Filling that gap is a snapshot: reading the current rows of a table and feeding them into the stream as if they'd just been written.¹

Initial snapshot — the whole table, once

The simplest mode: on first connect, read the entire table, emit every row, then switch to streaming the WAL. Controlled by snapshot.mode:

source.go:170 (generated default) · bob.json:36 (local override)

snapshot.mode = "never"     // generated default: skip the initial snapshot, rely on incremental
snapshot.mode = "initial"   // bob.json override: snapshot the whole table on first connect

Initial snapshots are simple but blunt — they read the whole table at once, which for a big table is slow and heavy, and if the connector restarts mid-snapshot it may start over. That's why the generated config defaults to never and leans on the smarter mode below.

Incremental snapshot — snapshot while streaming

The modern approach (Debezium's headline feature): snapshot existing rows while the WAL stream keeps running, in resumable chunks, triggered on demand. No stopping the stream, no starting over on restart.²

source.go:177,180

signal.data.collection          = "public.dbz_signals"   // the signal table
incremental.snapshot.chunk.size = "512"                  // rows per chunk

It's triggered by a signal: you INSERT a row into a special signaling table, and the running connector picks it up and starts chunking through the requested tables. In this repo the signal is written by a plain SQL INSERT:

internal/golibs/debezium/incremental_snapshot.go:39-62 (condensed)

func IncrementalSnapshot(ctx, db, signalTable string, data DataCollection) error {
    time.Sleep(snapshotWaitDuration)                 // wait for the Debezium task to be ready
    // wait until the replication slot is active (SELECT active FROM pg_replication_slots …)
    sql := fmt.Sprintf("INSERT INTO %s VALUES ($1, $2, $3)", signalTable)
    db.Exec(ctx, sql,
        id,                    // a unique id (sourceID + ULID)
        "execute-snapshot",    // the signal type
        data.String())         // {"data-collections": ["public.users", …]}
}

The killer feature: add a table without recreating the connector Here's why this matters so much. To capture a new table, you must both add it to the publication/include-list and backfill its existing rows. The naive way — delete and recreate the source connector — would drop the replication slot and lose the WAL position (Lesson 2). Instead: add the table to the include list, then fire an incremental snapshot signal for it. The running connector back-fills the existing rows in chunks while never dropping the slot. The code comment says it outright: "snapshot new added tables without recreating new source connector." This is wired into the control plane's --sendIncrementalSnapshot flag (Lesson 8).

A repo-history detail worth knowing There's an abandoned design here: subscription.go has a NATS-JetStream-driven snapshot handler that's entirely commented out (:108-142). The team started with a NATS-message-triggered flow, then simplified to the direct SQL INSERT above. So if you grep and find NATS snapshot code, know it's dead — the live path is IncrementalSnapshot. (A good reminder to check what runs, not just what exists.)

Debezium snapshots & signalling

How ad-hoc and incremental snapshots work, the signaling data collection, and the watermarking that lets a snapshot run alongside the live stream.

→ debezium.io — Incremental Snapshots
→ debezium.io — Sending signals · in-repo internal/golibs/debezium/incremental_snapshot.go

Check yourself (from memory)

Q1. Why does CDC need a snapshot at all?

A slot streams from its creation forward; existing rows must be snapshotted to reach the destination.

Q2. How is an incremental snapshot triggered here?

A direct SQL INSERT into the signal table with type execute-snapshot and a data-collections payload. (The NATS flow was abandoned.)

Q3. Why add a new table via incremental snapshot instead of recreating the source connector?

The slot is precious — dropping it loses changes. Incremental snapshot back-fills the new table's rows while the slot keeps running.

Recall: snapshots — initial vs incremental + the add-a-table trick.

why + both kinds + the trick, then reveal

Why: the slot streams only future WAL; existing rows need a snapshot. Initial (snapshot.mode=initial): read the whole table once on connect, then stream — simple but heavy, restarts from scratch. Generated default is never. Incremental: snapshot existing rows while streaming, in resumable chunks (chunk.size=512), triggered by a signal: INSERT into public.dbz_signals a row (id, 'execute-snapshot', {"data-collections":[…]}) — debezium.IncrementalSnapshot does this via direct SQL after the slot is active (the NATS flow is dead code). Killer use: add a new table to CDC by adding it to the include list + firing an incremental snapshot — without recreating the source connector (which would drop the slot). Wired to hephaestus's --sendIncrementalSnapshot.

🎯 Interview one-liner "How do you backfill existing data, or add a new table to CDC?" → "Snapshots. Initial reads the whole table on connect; incremental snapshots existing rows in chunks while the stream keeps running, triggered by a signal-table insert. That's how we add a table without recreating the source connector — recreating would drop the replication slot and lose changes."

Next: what the change events themselves look like — the Debezium envelope, the Avro topics, and how deletes appear. Ask me about chunking or the signal payload if you want more detail.

1. Debezium — snapshots.

2. Debezium — Incremental Snapshots, signalling. In-repo: internal/golibs/debezium/incremental_snapshot.go:16-72.