Lesson 14 · Senior CDC practice

Schema evolution, compatibility, and connector drift

How to reason about schema changes across source, topic, sink, and warehouse without pretending compatibility is somebody else’s problem.

Your win: explain how schema evolution becomes a multi-hop problem in CDC, and why generated connectors and field intersections help but do not remove the need for judgment.

Why schema lessons become more important at senior level

Early in the course, schema feels like structure: which columns exist, how the envelope looks, how the sink maps fields into the destination table. But later, when you think like the person reviewing changes instead of just reading them, schema stops being a static description and becomes a moving contract.

That shift matters because one table change rarely stays local in CDC. It can ripple through the source capture path, the event schema, the sink config, the destination table, and the warehouse logic that depends on it.

In plain English Plain English: changing one table safely is really changing several downstream contracts at once.

Why “the generator handles it” is not a complete answer

The repo does something genuinely useful here. The generation layer computes sink primary keys and fields.whitelist from the intersection of source and sink schemas. That removes a real class of drift and reduces manual connector mistakes.

But the strongest earlier lessons always separate what the tooling helps with from what it does not solve. Generation can align shapes mechanically. It cannot decide whether the meaning of the data still fits the destination, whether a warehouse transform now needs to change, or whether a renamed field quietly changed business semantics.

The drift rule A CDC schema change is safe only when the source, topic schema, connector config, and destination expectations still line up after the change.

What senior judgment adds

This is where senior review becomes different from implementation. The question is no longer just “Will the connector still run?” The better question is “Will the whole pipeline still mean the same thing after this change?”

That second question is the one that catches real drift. A pipeline can remain technically alive while becoming semantically wrong.

Backend use case In this repo, schema-aware generation in internal/platform/datapipeline/sink.go reduces connector drift, but warehouse SQL, destination semantics, and business meaning still need explicit review.
Common mistake Assuming “the generator will handle it” without checking whether downstream tables, joins, or analytical meanings still make sense.
Read this next

Schema evolution from source to sink

Use the schema-registry and JDBC docs with the repo map so compatibility stays tied to one real generated path instead of becoming an abstract platform topic.

→ Schema Registry
→ Repo CDC map

Check yourself (from memory)

Q1. What is the most senior way to think about schema evolution in CDC?

CDC turns local schema changes into pipeline-wide contract changes.
Why doesn’t generated connector config eliminate schema-evolution risk?
recall, then click to reveal
Because generation can align fields mechanically, but it cannot decide whether destination semantics, warehouse transforms, or business meanings still remain correct.
Want a “would this schema change drift the pipeline?” review drill? Ask me.

Sources. Schema Registry docs; repo CDC map.