# Kafka Cheat Sheet

Dense quick-reference for revision and interviews. Terms are defined in
[GLOSSARY.md](./GLOSSARY.md); our code specifics are in
[repo-kafka-map.md](./repo-kafka-map.md).

## The mental model in five lines
1. A **topic** is an append-only **log**, split into **partitions**.
2. Order is guaranteed **only within a partition**, addressed by **offset**.
3. A **producer** appends; the **key** decides the partition (same key → same partition → kept in order).
4. A **consumer group** splits the partitions among its members — **one partition per member** at a time.
5. Each group tracks its own **committed offset**, so groups read independently and can replay.

## Ordering & parallelism (the trade-off interviewers probe)
- More partitions → more parallelism (more consumers can work at once).
- But ordering holds only *within* a partition → to keep related events ordered, give them the **same key**.
- Max useful consumers in a group = number of partitions. Extra consumers sit idle.
- Repartitioning a live topic changes key→partition mapping → can break per-key ordering.

## Delivery semantics
| Semantic | Commit vs process | Outcome | In our repo |
|---|---|---|---|
| At-most-once | commit **before** process | may lose, never dup | `AlwaysCommit()` (loose) for a failed msg |
| At-least-once | commit **after** process | never lose, may dup | strict consumer (default) |
| Exactly-once | Kafka txns + idempotent producer | effect once | **not used** — app-level idempotency instead |

Rule of thumb: at-least-once **+ idempotent handler** ≈ exactly-once effect, far simpler. That is our design.

## Producer `acks`
| `acks` | Waits for | Durability | Speed |
|---|---|---|---|
| `0` | nothing | can lose on send | fastest |
| `1` / `RequireOne` | leader only | lose if leader dies pre-replication | middle — **our setting** |
| `all` | full ISR | strongest | slowest |

## Replication
- `replication factor = N` → N copies across brokers; tolerate `N-1` broker losses (with `acks=all`).
- **Leader** serves reads/writes; **followers** copy it; **ISR** = replicas caught up.
- Only an ISR member is promoted to leader without data loss.

## Retention vs compaction
- **Delete** (default): drop events older than `retention.ms` / bigger than `retention.bytes`.
- **Compact**: keep the latest event per **key** → topic becomes a "current state" changelog.
- Retention is about *keeping*, not *reading* — a consumer that never runs still ages out.

## Consumer group lifecycle
- Join/leave/crash → **rebalance** → partitions reassigned; consumption briefly pauses.
- Committed offset lives in internal topic `__consumer_offsets`.
- New group with no committed offset starts at `earliest` or `latest` (our `StartFromLastOffset()` = latest).

## Our repo — quick facts
```
Client        segmentio/kafka-go v0.4.39  (via internal/golibs/kafka)
Produce       PublishContext(ctx, topic, key, value)   // key → Hash balancer → partition
Consume       Consume(topic, groupID, opt, handler)     // handler returns (retryable bool, err)
Group id      {prefix}{service}.consumer-group.{topic}
Topic prefix  object_name_prefix, e.g. prod.tokyo.  → prod.tokyo.communication.email-sending
Payloads      JSON (no Avro / Schema Registry / proto-on-wire)
acks          RequireOne
Topic create  fink job upsert_kafka_topics  (NOT producer auto-create)
DLQ           none — retry via handler bool, else drop (loose) or stall (strict)
EOS           none — app-level idempotency (notification idempotentService)
Lanes         normal topic + .high-priority topic, separate groups, same pod
```

## Worked example — spike email
`SendEmail` gRPC → `TracedPublishContext(communication.email-sending, key=EmailID, json)`
→ spike `SendEmailConsumer` (group `spike.consumer-group.…`, `AlwaysCommit()`)
→ `SendEmailHandler.Handle` → SendGrid → status `PROCESSED`.
SendGrid fail → `(retryable=true)`; `ErrPayloadTooLarge` → `(false)` (don't retry a too-big email).

## Interview one-liners
- *Why partitions?* Parallelism + ordering. Ordering is per-partition; the key routes related events together.
- *Why consumer groups?* Scale out (split partitions) and fault tolerance (rebalance) while tracking progress via offsets.
- *At-least vs exactly once?* Commit-after-process gives at-least-once; make the handler idempotent to get exactly-once *effect*.
- *What is ISR?* The replicas caught up to the leader; `acks=all` waits for them; only they can be promoted safely.
- *Retention ≠ consumption.* Kafka deletes by age/size, not by "everyone read it."
- *Compaction?* Keep latest value per key → a changelog of current state.