# Resources — CDC / Streaming-ETL (Debezium + Kafka Connect)

High-trust sources only. In-repo anchors are the ground truth for *how this repo does it*; external
links are for *the general technique* the interview will test. Never cite parametric knowledge —
every non-trivial claim in a lesson links here.

## In-repo ground truth (the primary source for every lesson)
- **`reference/repo-cdc-map.md`** — the survey map: Kafka Connect deployment, the Debezium sources,
  the generated sink connectors, hephaestus (the control plane), the datapipeline abstraction,
  ksqlDB, and the warehouse. Start here.
- **Connect client golib:** `internal/golibs/kafkaconnect/connector.go` (the Connect REST API client).
- **Debezium golib:** `internal/golibs/debezium/` (`incremental_snapshot.go`, `subscription.go`).
- **The control plane:** `cmd/server/hephaestus/`, `internal/hephaestus/`, `features/hephaestus/`.
- **The pipeline abstraction:** `internal/platform/datapipeline/` (`source.go`, `sink.go`).
- **ksqlDB:** `internal/platform/ksqldb/init_migrate.go`.
- **The warehouse:** `internal/lmsdwh/`, `migrations/lmsdwh` (if present).
- **Connector configs:** `local/connectors/*.json`, `local/configs/services/kafka-connect-gen-sink-connector/`,
  `.../kafka-connect-source-connector/`.

## Change Data Capture & Debezium (official)
- **Debezium — PostgreSQL connector** — https://debezium.io/documentation/reference/stable/connectors/postgresql.html —
  the source connector: logical decoding, replication slots, LSN, snapshots, change-event shape.
- **Debezium — logical decoding output plugins** — https://debezium.io/documentation/reference/stable/postgres-plugins.html —
  `pgoutput` vs `wal2json`/`decoderbufs`, `wal_level=logical`, publications.
- **Debezium — sending signals / signalling** — https://debezium.io/documentation/reference/stable/configuration/signalling.html —
  the signaling data collection (`dbz_signal`) that triggers ad-hoc & incremental snapshots.
- **Debezium — Incremental Snapshots (blog)** — https://debezium.io/blog/2021/10/07/incremental-snapshots/ —
  chunked, resumable snapshots of existing data with the stream running (watermark method).
- **Debezium — the outbox pattern** — https://debezium.io/documentation/reference/stable/integrations/outbox.html —
  reliable "publish an event as part of the DB transaction" (this repo has an outbox golib too).

## Kafka Connect — the framework (official / high-trust)
- **Kafka Connect (Confluent)** — https://docs.confluent.io/platform/current/connect/index.html —
  connectors, tasks, workers, converters, SMTs, standalone vs distributed.
- **Kafka Connect 101 (Confluent Developer)** — https://developer.confluent.io/courses/kafka-connect/ —
  the best free course; the REST API lesson https://developer.confluent.io/courses/kafka-connect/rest-api/.
- **Apache Kafka — Connect docs** — https://kafka.apache.org/documentation/#connect — the upstream reference.
- **Kafka Connect JDBC Sink connector** — https://docs.confluent.io/kafka-connectors/jdbc/current/sink-connector/overview.html —
  the sink into a relational warehouse; `insert.mode`, upsert, `pk.mode`/`pk.fields`, DLQ.
- **JDBC Sink primary keys (Robin Moffatt)** — https://rmoff.net/2021/03/12/kafka-connect-jdbc-sink-deep-dive-working-with-primary-keys/ —
  the practical guide to upsert + PK handling.

## Serialization & schema (official)
- **Confluent Schema Registry** — https://docs.confluent.io/platform/current/schema-registry/index.html —
  centralized schemas + compatibility; how connectors register schemas.
- **Avro serdes** — https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/serdes-avro.html —
  the `AvroConverter`, wire format, schema IDs.

## ksqlDB — stream processing (official)
- **ksqlDB — streams & tables** — https://developer.confluent.io/courses/ksqldb/streams-and-tables/ —
  the two first-class types; stream = event log, table = current state per key.
- **ksqlDB — queries (push vs pull)** — https://docs.confluent.io/platform/current/ksqldb/concepts/queries.html —
  `EMIT CHANGES` (push) vs point-in-time (pull) against materialized views.

## Concepts & wisdom (technique the interview tests)
- **Log-based vs query-based CDC** — the Debezium FAQ + blog https://debezium.io/blog/ — why reading
  the WAL beats polling (no missed changes, deletes, low load).
- **Kleppmann — "Turning the database inside out"** — https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ —
  the mental model: the log as the source of truth, materialized views downstream.
- **The dual-write problem & the outbox** — https://debezium.io/blog/2019/02/19/reliable-microservices-data-exchange-with-the-outbox-pattern/ —
  why CDC/outbox beats "write to DB and publish to Kafka" separately.

## Community (wisdom — test your understanding on real people)
- **Confluent Community Forum** — https://forum.confluent.io/ — Kafka Connect & Debezium categories,
  high-signal Q&A on connector config, upsert, DLQs.
- **Debezium Zulip / mailing list** — https://debezium.io/community/ — the maintainers answer here.
- **r/apachekafka** — https://www.reddit.com/r/apachekafka/ — CDC design and pipeline debates.

## The interview angle
- Staples worth rehearsing against this repo's pipeline: what CDC is and why log-based; the roles of
  source vs sink connectors; snapshots (initial vs incremental) and why they matter; delivery
  semantics (at-least-once) and idempotent sinks (upsert); schema evolution; the dual-write problem
  and outbox; and a control-plane design (why generate/reconcile connectors rather than hand-write
  248 of them). Ground each in a real config/file here, not a toy.