# Glossary — CDC / Streaming-ETL (this repo)

The canonical vocabulary. Opinionated: where the repo has a house term, that wins; the general
industry term is noted so you can map to the interview. Adhere to these names in every lesson.

---

### CDC (Change Data Capture)
Turning a database's row-level changes (insert/update/delete) into a stream of change events, as they
happen. Here it's **log-based** CDC: read the write-ahead log, don't poll. The pipeline's front door.

### log-based CDC (vs query-based)
Reading changes from the DB's replication log (Postgres WAL) instead of polling tables with
`WHERE updated_at > …`. Log-based catches *every* change including deletes, adds ~no load, and can't
miss updates between polls. This repo is entirely log-based (Debezium).

### the three stacks
This repo runs CDC three separate times, with three Connect clusters / registries / ksqlDB servers:
(1) **cross-service data-sync** (replicate tables between services' Postgres — the biggest),
(2) the **KEC warehouse** (CDC + ksqlDB → warehouse), (3) **lmsdwh** (LMS analytics DWH, with chained
CDC). Never say "the cluster" — say which stack.

### Debezium
The open-source CDC engine. Here, the **Postgres source connector** (`io.debezium.connector.postgresql.PostgresConnector`)
reads the WAL and emits a change event per row change. Runs as a Kafka Connect source connector.

### WAL (write-ahead log)
Postgres's durable, ordered log of every change, written before the change hits the table. CDC reads
it. Positions are **LSNs** (Log Sequence Numbers).

### logical decoding / `pgoutput`
Postgres's mechanism for turning raw WAL into a logical stream of row changes. `pgoutput` is the
native output plugin this repo uses (`plugin.name = pgoutput`) — no external plugin to install.
Requires `wal_level = logical`.

### replication slot
A named, server-side bookmark that **retains WAL** until CDC has consumed it — so Debezium can go
down and resume without missing changes. Named `{env}_{org}_{db}` (e.g. `local_manabie_bob`).
`slot.drop.on.stop = false` so it survives restarts. If a slot's connector is deleted, the slot is
dropped and its position lost — which is why the repo **refuses to delete source connectors**.

### publication
A Postgres object that declares *which tables* logical replication carries. `publication.name =
debezium_publication`, and — repo-specific — **`publication.autocreate.mode = disabled`**: the
publication is created and altered by **SQL migrations**, not by Debezium.

### snapshot
Loading the *existing* rows of a table into the stream (the WAL only has *future* changes). Two kinds
below.

### initial snapshot
On first connect, read the whole table once, then switch to streaming the WAL. `snapshot.mode`
controls it (`initial` vs `never`).

### incremental snapshot
Snapshot existing data **while the stream keeps running**, in resumable **chunks** (`chunk.size =
512`), triggered on demand by a **signal**. This is how a new table is added to CDC **without
recreating the source connector** — the killer feature here.

### signal / signal table (`dbz_signals`)
The row you INSERT into a Debezium signaling table (`signal.data.collection = public.dbz_signals`) to
tell the running connector to do something — here, `'execute-snapshot'` with a `{"data-collections":
[…]}` payload. In this repo the signal is written by a **direct SQL INSERT**
(`debezium.IncrementalSnapshot`), not the abandoned NATS flow.

### change event / the Debezium envelope
The message Debezium emits per change: a JSON/Avro record with `before`, `after`, `op`
(c/u/d/r=read), and `source` metadata. The sink's `unwrap` SMT flattens it to just the `after` row.

### tombstone
A message with a **null value** emitted after a delete (`tombstones.on.delete = true`) so log-compacted
topics can garbage-collect the key. This repo's sinks **drop** tombstones (deletes aren't propagated).

### Kafka Connect
The framework that runs connectors. **Distributed mode** here (a customized Debezium image), config
stored in Kafka topics, managed over a **REST API** on `:8083`. Both Debezium sources and JDBC sinks
are Connect connectors.

### connector / task / worker
A **connector** is a job definition (source or sink); it splits into **tasks** (the units that
actually copy data); **workers** are the processes that run tasks, balanced across the cluster.

### converter / Schema Registry / Avro
A **converter** serializes a change event to/from the bytes on the Kafka topic. This repo uses the
Confluent **AvroConverter** with a **Schema Registry** (`cp-schema-registry:8081`) — schemas are
centralized and versioned; the message carries a schema ID, not the schema.

### SMT (Single Message Transform)
A lightweight per-message transform in the connector config. The two here: **`unwrap`**
(`ExtractNewRecordState` — flatten the envelope to the row) and **`route`** (`RegexRouter` — rewrite
the topic/table name). Plus a repo-authored Java `CustomFilter` and Confluent `Filter$Value` for
tenant filtering.

### sink connector (JDBC)
The connector that writes a Kafka topic INTO a database: Confluent `JdbcSinkConnector`. Here it does
an **idempotent upsert** (`insert.mode = upsert`, `pk.mode = record_value`, `pk.fields = …`) into
another service's Postgres or the warehouse.

### upsert
Insert-or-update on a primary-key conflict — makes the sink **idempotent**, so re-processing a
message (at-least-once delivery) doesn't duplicate or corrupt. `insert.mode = upsert`.

### `fields.whitelist` (the intersection)
The columns a sink writes = the **intersection** of the source table's and the sink table's columns
(computed by the generator from real DB schemas). Keeps two independently-evolving schemas from
breaking the sink.

### the connector-generation system
This repo doesn't hand-write ~248 connectors — it **generates** them. Declarative YAML defs
(`generated_connectors/<db>.yaml`: source DB + envs + orgs + `pipelines`) fan out **env × org × table
× sink** into one JSON each, via the `datapipeline` package + `gen-data-pipeline`. Source-centric
fan-out; **bob is the hub**.

### hephaestus (the control plane)
A **job runner** (not a long-lived server) that reconciles the Connect clusters. It reads the
generated connector JSON (the desired state, in git), diffs against the live cluster over the REST
API, and **creates/updates/deletes to converge** — with a **delete-guard** that only removes JDBC
sink connectors (never sources or ksqlDB connectors).

### git-as-desired-state (reconcile loop)
The control-plane model: there is **no database of connectors**. The connector JSON files in the repo
*are* the desired state; hephaestus's job is to make the live cluster match. (Same idea as
Kubernetes/GitOps, applied to Kafka Connect.)

### ksqlDB
A stream-processing engine over Kafka: **streams** (event logs) and **tables** (current state per
key), queried with SQL. Here it joins/aggregates CDC topics into `dim_*` shapes and declares its own
sink connectors. "ksqlDB migrations" = the Confluent `ksql-migrations` CLI (versioned SQL).

### stream vs table (ksqlDB)
A **stream** is an unbounded series of events (append-only); a **table** is the latest value per key
(a materialized view). Both are built from Kafka topics — same data, different interpretation.

### the data warehouse (lmsdwh) / star schema
The analytics destination: a **star schema** of `dim_*` (dimensions — user, student, course…) and
`fact_*` (facts — enrollments, assessment results…) tables, for BI/reporting. The medallion arc:
raw CDC refined into query-friendly "gold" tables.

### chained CDC
CDC feeding CDC: services → lmsdwh Postgres → (a *second* Debezium source) → partner DWHs. Two
publications/slots/registries in series. Unusual and repo-specific.

### the dual-write problem / outbox
Writing to the DB and publishing to Kafka as two separate steps can partially fail (one succeeds, one
doesn't) → data drift. CDC (and the **outbox pattern**) fix it by deriving the event *from the
committed DB change* — one source of truth. (This repo also has an `outboxevents` golib.)

### at-least-once + idempotent
The pipeline's delivery guarantee: a message may be delivered more than once (at-least-once), and the
**upsert sink** makes that safe (idempotent). No exactly-once/transactional config is used.