# Mission — CDC / Streaming-ETL in *this* repo

## Who this is for
A backend engineer in this monorepo who wants to understand how data **moves between services and
into the warehouse** — Change-Data-Capture with Debezium, the Kafka Connect framework, the ~248
generated connectors, and the **hephaestus** control plane — and to be **interview-ready** on
streaming-ETL technique.

Strong backend maturity (this is far from their first course; the Kafka course is already done).
**Newer to this repo's data pipeline**: log-based CDC, the Debezium Postgres source, replication
slots/publications, snapshots, the JDBC sink connectors, the connector-generation system, and how
hephaestus reconciles all of it.

## The mission
> *"Teach me the CDC / streaming-ETL pipeline — Debezium + Kafka Connect and everything around it —
> so I can understand how data flows across the platform and prepare for the interview."*

Two concrete targets:
1. **Catch-up fluency** — trace how a single row change in one service's Postgres ends up in another
   service's DB (and in the warehouse), naming every hop and the config that drives it.
2. **Interview readiness** — explain log-based CDC, source vs sink connectors, snapshots, delivery
   semantics + idempotent sinks, schema evolution, the dual-write/outbox problem, and why you'd
   *generate and reconcile* connectors rather than hand-write hundreds.

## The lead focus (chosen by the learner)
**Balanced, end-to-end** — follow one row's change through *every* stage: source (Debezium CDC) →
Kafka topic → (optional ksqlDB transform) → sink connector → destination DB / warehouse. No single
stage dominates; the *pipeline* is the subject.

## The signature to feature (chosen by the learner)
**hephaestus + the connector-generation system, prominently.** This repo does not hand-write its
~248 connectors — it **generates** them from declarative YAML (the `datapipeline` package) and
**reconciles** them git-as-desired-state via hephaestus jobs. That control-plane design is what makes
this repo's CDC setup distinctive and interview-worthy, and it gets first-class treatment.

## What "done" looks like
- Can name the **three CDC stacks** (cross-service data-sync, the KEC warehouse, lmsdwh) and say what
  each is for.
- Can explain **log-based CDC**: the Postgres WAL, `pgoutput` logical decoding, replication slots,
  publications, and why it beats polling.
- Can distinguish **initial vs incremental snapshots** and explain the signal-table trick that adds a
  new table to CDC *without recreating the source connector*.
- Can read a **sink connector** config: JDBC upsert, `pk.fields`, the `unwrap`+`route` SMT chain, and
  why deletes are dropped.
- Can explain the **generation + reconcile** loop: declarative defs → generated JSON → hephaestus
  converges the live Connect cluster to match.
- Can trace **`bob.public.users` → `auth.users`** end to end, and defend the delivery semantics
  (at-least-once + idempotent upsert).

## Non-goals
- Not the Kafka *log* course (that exists) — this is the pipeline *on top of* Kafka.
- Not a full data-warehouse/BI-modeling course — the warehouse (lmsdwh) is the *destination*, taught
  as far as the pipeline needs.
- Not exhaustive of all three stacks' every connector — the mechanics generalize; bob→auth is the
  worked example.