Lesson 5 · Kafka Connect & the sink

The Kafka Connect framework

The runtime that runs both the Debezium sources and the JDBC sinks — connectors described by config, not code, managed over a REST API.

Your win: name the Kafka Connect building blocks (connector, task, worker, converter, SMT), explain distributed mode and the REST API, and describe how this repo runs Connect.

Config, not code

Kafka Connect is a framework for streaming data between Kafka and other systems. Its whole point: you move data by writing a JSON config, not a program.¹ Everything you've seen so far — the Debezium source, the JDBC sink — is a Connect connector, just a block of config posted to a cluster. Same framework, two directions: a source connector reads into Kafka, a sink connector writes out.

The building blocks

Thing	What it is
Connector	a job definition (the JSON) — "capture bob's tables" or "sink this topic to auth"
Task	the unit that actually copies data; a connector splits into one or more tasks (`tasks.max`)
Worker	a process that runs tasks; the cluster balances tasks across workers
Converter	serializes each record to/from the bytes on the topic — here the Avro converter (Lesson 4)
SMT	a Single Message Transform — a lightweight per-message tweak in the config (Lesson 6's `unwrap`/`route`)

Distributed mode + the REST API

Connect runs in distributed mode here: multiple workers share a group, coordinate, and rebalance tasks when one dies — resilient and scalable (vs standalone = one process, for dev).² The cluster's own state (which connectors exist, their offsets, their status) lives in Kafka topics:

local/docker-compose.infra.yaml:204-214 (condensed)

CONFIG_STORAGE_TOPIC = my_connect_configs      # connector configs
OFFSET_STORAGE_TOPIC = my_connect_offsets      # where each source/sink is up to
STATUS_STORAGE_TOPIC = my_connect_statuses     # RUNNING / FAILED per connector+task
CONNECT_CONFIG_PROVIDERS = file                # ${file:…:password} pulls secrets from a file

You manage it all through the REST API on :8083 — create, update, delete, and check status of connectors.² That REST API is exactly what this repo's Go client talks to:

internal/golibs/kafkaconnect/connector.go:23-31 (the client interface)

type ConnectClient interface {
    GetConnector / CreateConnector / UpdateConnectorConfig / DeleteConnector / ListConnectors …
}

This repo runs a customized Debezium image — and three clusters The Connect image isn't vanilla Confluent — it's a customized Debezium Connect image (customized_debezium_connect:2.4.1.Final.…) baked with the plugins this repo needs: the Debezium Postgres source, the Confluent JDBC sink, and a repo-authored custom SMT (Lesson 10). And remember the three stacks (Lesson 1): there are three Connect clusters (data-sync, KEC warehouse, lmsdwh), each with its own REST API, schema registry, and ksqlDB. "The cluster" is always one of three.

Where the pieces you've met fit Debezium (Lessons 2–4) is a source connector running on this framework; its converter is the Avro converter; its route transform is an SMT. The JDBC sink (Lesson 6) is a sink connector on the same cluster. Kafka Connect is the stage; the connectors are the actors.

Kafka Connect concepts & the REST API

The best free walkthrough of connectors/tasks/workers/converters/SMTs and managing them over the REST API.

→ Confluent Developer — Kafka Connect 101
→ Confluent — Kafka Connect · the REST API

Check yourself (from memory)

Q1. How do you define a connector in Kafka Connect?

Config, not code: a connector is a JSON block. Both Debezium sources and JDBC sinks are just configs on the cluster.

Q2. In distributed mode, where does the cluster keep its own state?

my_connect_configs/_offsets/_statuses topics — so any worker can pick up any connector after a rebalance.

Q3. What does the Go ConnectClient talk to?

Get/Create/Update/Delete/List connectors all go through the REST API — the only management interface for a distributed cluster.

Recall: what Kafka Connect is + the building blocks.

config-not-code + 5 pieces + distributed/REST, then reveal

Kafka Connect = a framework to stream data between Kafka and other systems via JSON config, not code. A source reads into Kafka (Debezium); a sink writes out (JDBC). Building blocks: connector (the config/job) → tasks (units that copy data, tasks.max) run by workers (processes, balanced across the cluster); converter (serialize to/from topic bytes — Avro); SMT (per-message transform). Distributed mode: workers share a group + rebalance; cluster state in Kafka topics (my_connect_configs/_offsets/_statuses). REST API :8083 = the only management interface (what the Go ConnectClient uses). Repo runs a customized Debezium image (source + JDBC sink + custom SMT plugins), in three clusters.

🎯 Interview one-liner "What is Kafka Connect?" → "A framework for moving data in and out of Kafka with configuration instead of code. A connector is a JSON job that splits into tasks run by workers; converters handle serialization, SMTs do light per-message transforms. We run it distributed — cluster state in Kafka topics, managed over the REST API — with both Debezium sources and JDBC sinks on it."

Next: the sink up close — JDBC sink connectors, read line by line, including the upsert and the two SMTs that make the row land correctly. Ask me about tasks or rebalancing if you want more depth.

1. Confluent — Kafka Connect. In-repo: internal/golibs/kafkaconnect/connector.go:23-31.

2. Confluent — Connect REST API. In-repo: local/docker-compose.infra.yaml:193-220.