Lesson 12 · The repo in practice

NATS vs Kafka & ops

Why this repo runs two buses, why it's moving some flows between them, how NATS is operated — and the whole course in one view.

Your win: articulate when to reach for NATS vs Kafka, explain the live migration, know the ops essentials — and recite the whole pipeline.

NATS vs Kafka — the division of labour

NATS JetStreamKafka
Strengthultra-low latency, lightweight, simple opsdurability, huge throughput, long retention, replay
Used here forinternal service events & fan-out (user/staff/usergroup upserts, push-notification requests), the audit firehosedurable pipelines: CDC (Debezium), the outbox, email, bulk jobs, DWH sync
Retentioninterest (keep while a consumer needs it)time/size-based log (replayable history)
NamingStaff.Upserted (dotted PascalCase){team}.{topic}
The one-line rule Reach for NATS when you want a fast, simple, at-least-once event that a few known services react to and then forget. Reach for Kafka when you need a durable, replayable log, very high throughput, or a data pipeline (CDC). The repo uses NATS for "something happened, tell the interested services" and Kafka for "capture this durably and stream it."1

The live migration: NATS → Kafka

You've seen the commented-out subscribers (Lessons 9–10). The repo is migrating some flows from NATS to Kafka, gated by an Unleash flag (Communication_Notification_Use_Kafka_Instead_Of_NATs). Because the domain handler is transport-agnostic (Lesson 10), migrating a flow means swapping the transport shell, not rewriting the logic:

before: event ──NATS subject──▶ [NATS transport shell] ──▶ ProcessX (domain handler) after: event ──Kafka topic────▶ [Kafka transport shell] ─▶ ProcessX (SAME domain handler)

Why move a flow at all? When it outgrows NATS's sweet spot — it needs durable replay, higher throughput, or to join the CDC/data pipeline. The Unleash flag lets the team flip a flow over (and back) safely. So "which bus?" is sometimes a runtime decision, and the design makes it cheap.

Operating NATS

The server

JetStream with file storage (store_dir, 4 GiB mem / 10 GiB file), client port 4223, monitoring on 8223, a 3-replica StatefulSet in prod (1 locally).

Per-service ACLs

nats.conf gives each service its own account that may only publish its own subjects (and subscribe to its inboxes). fink is unrestricted (it provisions everything). A real security boundary.

Provisioning & tests

Streams via the fink job; consumers at subscribe time. Metrics via OpenCensus views + a disconnect gauge. Subscribers are unit-tested against a mock JetStreamManagement (testify).

🏛️ The whole course, in one view

CORE NATS & THE MODEL (Part 1) core NATS: pub/sub, at-most-once, no persistence · JetStream: streams + consumers, at-least-once subjects (dotted, wildcards * / >) · queue groups (load-balance) · streams (interest retention, fink) consumers (durable push queue-group; pull for the firehose) THE GOLIB (Part 2) lazy connect (rsc.NATS, sync.Once) · publish wraps DataInMessage{payload, resource_path, trace} + MsgId(ULID) dedup handleMsg: unmarshal → RE-INJECT TENANCY (ctx) → span → cb → ack/retry/drop ← RLS survives the async hop delivery: at-least-once (AckExplicit) ; Ack-only (no Nak/Term) ; MaxDeliver→drop (no DLQ) ; dedup+idempotent=effectively-once YOUR SERVICES (Part 3) conversationmgmt: subscriber-only (User.Created, Staff.Upserted, UserGroup.Upserted) → sync Agora chat notification: owns the notification stream ; convo ──Notification.Created──▶ notification → FCM ; transport-agnostic handler activity-log firehose: interceptors publish every request → zeus PULL consumer batches → partitioned table NATS vs KAFKA: NATS = fast internal events/fan-out ; Kafka = durable pipelines/CDC ; live NATS→Kafka migration (Unleash)
The five ideas that carry the interview 1. Core vs JetStream — at-most-once pub/sub vs at-least-once streams+consumers. 2. Streams & consumers — a stream stores subjects (interest retention); a consumer is a durable cursor with its own ack config. 3. Tenancy survives the hop — the envelope carries resource_path, re-injected on the consumer, so RLS works on async events. 4. At-least-once + idempotent — explicit ack + dedup + idempotent handlers = effectively-once. 5. NATS vs Kafka — fast internal events vs durable pipelines, with a transport-agnostic handler that makes migration a swap.
Read this next

NATS & Kafka compared

The trade-off in depth — latency, ops, delivery, use cases — to cement the "which bus" instinct.

→ Synadia — NATS & Kafka compared
→ docs.nats.io — Compare NATS · in-repo local/configs/services/nats-jetstream/nats.conf

Check yourself (from memory)

Q1. When should you reach for NATS over Kafka in this repo?

NATS = fast, simple, fan-out events with interest retention. Kafka = durable replayable pipelines (CDC, outbox, DWH).

Q2. Why can a flow migrate from NATS to Kafka without a rewrite?

The business logic lives in a shared handler; swapping the transport (behind an Unleash flag) keeps it intact.

Q3. What do the per-service NATS account ACLs enforce?

Each service's account can only publish its own subjects (subscribe to inboxes) — a security boundary. fink is unrestricted.
Recall: NATS vs Kafka + the migration + ops.
the split + migration + ops, then reveal
NATS vs Kafka: NATS = ultra-low-latency, simple, internal service events / fan-out (interest retention, dotted subjects) + the audit firehose; Kafka = durable, high-throughput, replayable pipelines (CDC, outbox, DWH; {team}.{topic}). Rule: NATS = "tell interested services, then forget"; Kafka = "capture durably + stream". Migration: active NATS→Kafka (Unleash flag …Use_Kafka_Instead_Of_NATs); the transport-agnostic handler makes it a shell swap, not a rewrite. Ops: JetStream file storage (4GiB/10GiB), client :4223 / monitoring :8223, 3-replica StatefulSet (1 local); per-service ACLs (publish own subjects only; fink unrestricted); streams via fink; mock JetStreamManagement for tests.
🎓🏁 Course complete — NATS JetStream Twelve lessons: core NATS & the JetStream model (pub/sub, subjects, streams, consumers) → the golib (connection, publishing, the handleMsg heart, delivery) → the repo in practice (your conversationmgmt & notification subscribers, the publish link, the firehose, NATS vs Kafka). You can now open any subscriber you own and read it fully — and name what the repo really does: tenancy through the envelope so RLS survives the hop, Ack-only redelivery with no DLQ, the transport-agnostic handlers, and the live migration. Interview-grade fluency in the eventing your services live on.
That's the build. The one thing that would prove it stuck is a retention check: ask me to run a mock interview across this course (core → the golib → your services, cold, no peeking) and I'll write the first real learning records showing where you're solid and where to review. You've built the knowledge; let's confirm it's yours. Ask me to run it.

1. Synadia — NATS & Kafka compared. In-repo: local/configs/services/nats-jetstream/nats.conf, internal/notification/CLAUDE.md (the migration flag).