Lesson 1 · Foundations

What Kafka is, and why it exists

The single idea underneath everything: an append-only log.

Your win for this lesson: explain in one breath what Kafka is, and why your SendEmail API publishes an event to Kafka instead of just calling SendGrid directly.

Start with a problem you already have

When a client calls spike's SendEmail gRPC endpoint, the email eventually goes out through SendGrid. The naive design is: the API handler calls SendGrid right there, synchronously, and waits.

That design hurts the moment reality shows up. SendGrid is slow or down → your API is slow or down. A burst of 10,000 emails → 10,000 requests hammering SendGrid at once. The API pod restarts mid-send → those emails are simply lost. The sender and the sender-of-emails are welded together.

BEFORE (coupled) AFTER (decoupled through Kafka) SendEmail ──▶ SendGrid SendEmail ──▶ [ Kafka log ] ──▶ spike ──▶ SendGrid (waits, fails together) append & return read at own pace

Kafka's answer: put a log in the middle

Kafka is a distributed, durable, append-only log that sits between the thing producing events and the thing reacting to them.¹ A producer appends an event; a consumer reads it later, at its own pace. Neither has to be online at the same time, and the event is written to disk and replicated, so it survives restarts.

This is the whole trick. Jay Kreps — who co-created Kafka — argues the humble log is the unifying abstraction for real-time data: a series of records you can only append to, ordered by time.² Everything else in Kafka is built on top of that primitive.

The one idea A Kafka topic is a durable, append-only log of events. Producers append; consumers read. The log decouples them in time (read later), space (different services), and rate (read as fast as you can).

Your code, exactly

This is precisely what spike does. The gRPC handler doesn't call SendGrid — it publishes an event and returns immediately:

internal/spike/modules/email/controller/grpc/email_modifier_send_email.go:47

// SendEmail handler — appends an event to the log, then returns.
topic := TopicByPriority(svc.resolvePriority(ctx, clientID))
err := svc.kafka.TracedPublishContext(ctx, spanName,
        topic,                 // communication.email-sending
        []byte(email.EmailID), // the key (more on this in Lesson 2)
        payload)               // the SendEmailEvent, as JSON

Somewhere else, whenever it's ready, the spike consumer reads that event and does the slow work of actually talking to SendGrid:

internal/spike/modules/email/application/consumers/send_email_handler.go:46

// SendEmailHandler.Handle — reads the event later, sends the email.
event := json.Unmarshal(data) // SendEmailEvent
sendViaSendGrid(event)
updateStatus(PROCESSED)

Because the log is in the middle: the API stays fast (it only appends), a burst becomes a backlog the consumer drains steadily, and a crash loses nothing — the event is still in the log when the consumer comes back. That is why Kafka is there.

Anchor The topic is communication.email-sending. Producer: SendEmail gRPC. Consumer: spike's SendEmailHandler. Keep this flow in your head — we reuse it in every lesson. Full map: repo-kafka-map.md.

Words you now own

Event (an immutable fact) · Topic (the log) · Producer (appends) · Consumer (reads). Tight definitions live in the glossary — the course uses those exact words from here on.

Read / watch this next

Apache Kafka 101 — Confluent Developer

The best beginner on-ramp, from the people who built Kafka. Watch the first two short videos (“Events” and “Topics”). Then, for the deeper why, skim Jay Kreps' essay on the log.

→ developer.confluent.io/courses/apache-kafka
→ “The Log” — Jay Kreps

Check yourself (from memory)

Q1. What is a Kafka topic, most precisely?

A topic is a log: events are appended and stay there (until retention), and reading does not remove them. That's what lets many consumers read the same events independently, and replay them.

Q2. Why does SendEmail publish an event rather than call SendGrid directly?

The log decouples them: the API returns fast after appending, the consumer drains the work at its own rate, and a crash loses nothing. Kafka never talks to SendGrid — spike's consumer does.

In one sentence: what does the log give you that a direct API call doesn't?

recall, then click to reveal

Decoupling in time, space, and rate — plus durability. The producer doesn't wait for the slow work, the consumer reads whenever it's ready, and events survive restarts because they're written to disk and replicated.

Something fuzzy — “append-only,” “durable,” why not just a normal queue? Ask me. I'm your teacher for this course.

1. Apache Kafka Documentation — Design.

2. Jay Kreps, “The Log: What every software engineer should know…”, LinkedIn Engineering, 2013.