# Apache Kafka Resources

Curated, high-trust sources for this workspace. Explainers in the lessons draw
from here — not from memory. Grouped **Knowledge** (learn the ideas), **Wisdom**
(test them against real people), and **Repo** (our own code, the ground truth for
this mission).

## Knowledge

### Start here (beginner, interview-shaped)
- [Apache Kafka 101 — Confluent Developer (free course, Tim Berglund)](https://developer.confluent.io/courses/apache-kafka/events/)
  13-part video + text course from the company founded by Kafka's creators. The
  single best beginner on-ramp. **Use for:** first pass on every core concept —
  events, topics, partitions, brokers, replication, producers, consumers.
- [_Gently Down the Stream_ — Mitch Seymour](https://www.gentlydownthe.stream/)
  A picture book about otters that quietly teaches the Kafka mental model. Read it
  free online. **Use for:** building intuition before the vocabulary; great when a
  concept won't "click."

### Primary reference (authoritative)
- [Apache Kafka Documentation — Design section](https://kafka.apache.org/documentation/#design)
  The canonical source. Terse but exact. **Use for:** the precise definition of the
  log, partitions, replication/ISR, delivery semantics, message format. When a
  lesson makes a claim, this is what it's checked against.
- [Confluent Documentation — Kafka Design: Consumers, Groups & Offsets](https://docs.confluent.io/kafka/design/consumer-design.html)
  **Use for:** consumer group mechanics, the `__consumer_offsets` topic, the pull
  model, rebalancing.
- [Confluent Developer — Kafka Internal Architecture course](https://developer.confluent.io/courses/architecture/get-started/)
  **Use for:** a level deeper than 101 — how the broker, controller, and replication
  actually work. Good for the "explain the internals" interview questions.

### Book (deep dive, free)
- [_Kafka: The Definitive Guide_, 2nd ed. — Shapira, Palino, Sivaram, Petty (free PDF via Confluent)](https://www.confluent.io/resources/ebook/kafka-the-definitive-guide/)
  Written by engineers who built Kafka. **Use for:** the definitive treatment of
  reliability (`acks`, replication), consumer internals, and exactly-once. Read the
  relevant chapter *after* the matching lesson, not before.

### The "why it exists" essay (conceptual gold)
- [_The Log: What every software engineer should know about real-time data's unifying abstraction_ — Jay Kreps (2013)](https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying)
  By Kafka's co-creator. The idea that the log is the primitive everything else is
  built from. **Use for:** the "why does Kafka look like this?" interview answer that
  makes you sound like you *get it*, not just memorised it.

### Go client (matches our codebase)
- [`segmentio/kafka-go` — GitHub README](https://github.com/segmentio/kafka-go) ·
  [pkg.go.dev reference](https://pkg.go.dev/github.com/segmentio/kafka-go)
  **Use for:** how `Reader` (consumer groups via `GroupID`), `Writer`, offset
  commits, and balancers map to Kafka concepts in Go. *(Confirm this is the client
  our repo uses — see Repo below.)*

## Wisdom (Communities)

- [Confluent Community Forum](https://forum.confluent.io/)
  Moderated, staffed by Confluent engineers. **Use for:** "is my mental model right?"
  questions and design critique.
- [r/apachekafka](https://www.reddit.com/r/apachekafka/)
  Active, reasonably high-signal. **Use for:** real-world gotchas, interview-experience
  threads, "why did my consumer do X" stories.
- [Apache Kafka Users mailing list](https://kafka.apache.org/contact)
  The upstream community. **Use for:** authoritative answers on Kafka behaviour when
  docs are ambiguous.

## Repo (ground truth for this mission)

Our own Kafka usage is the most relevant "resource" of all — it's what the mission
is anchored to. Concrete file paths, topics, consumer groups, and the wrapper library
are captured in [reference/repo-kafka-map.md](./reference/repo-kafka-map.md)
*(generated from a scan of `internal/notification`, `internal/spike`, and
conversationmgmt — pending the codebase map, then filled in).*

## Gaps
- No repo-specific runbook yet for diagnosing consumer lag in our services — if the
  mission shifts toward ops, this is the first gap to fill.
- Interview question banks vary wildly in quality; none trusted enough to list yet.
  The lessons build their own recall quizzes from the authoritative sources above.