Lesson 10 · Applied to your services

The priority-lane pattern

Kafka has no message priorities. So how does urgent email jump the queue?

Your win: explain why Kafka can't prioritise messages within a topic, and how our services get priority anyway with two topics + two consumer groups — plus the trade-off that choice makes.

The constraint: a partition is strictly FIFO

A partition is an append-only log read in offset order (Lesson 2). There is no "read the important ones first" — Kafka has no native message priority.¹ If urgent and bulk emails share one topic, an urgent email queued behind 50,000 bulk ones waits for all 50,000. A "priority" field in the payload wouldn't help: the consumer still receives them in log order.

Why the obvious ideas fail One topic + priority field? No — order is fixed by offset, not by a field. Just add partitions? No — that adds parallelism, but urgent and bulk still interleave across those partitions. Neither isolates urgent work from a bulk backlog.

The pattern: separate lanes

The simplest working answer — and the one our services use — is a topic per priority, each drained by its own consumer group, running concurrently.¹ The producer routes by priority; the two lanes never share a queue, so a bulk backlog on one lane can't delay the other.

producer (routes by priority) ├─▶ communication.email-sending ─▶ group A ─┐ └─▶ communication.email-sending.high-priority ─▶ group B ─┤ same pod, two goroutines, high lane stays fast even if the normal lane has a 50k backlog running concurrently

In spike, one consumer object starts both lanes — high priority first, then normal — each as its own running loop:

internal/spike/modules/email/controller/kafka/send_email_consumer.go:21,42

ConsumeHighPriority()  // topic .high-priority, its own group, AlwaysCommit
Consume()              // topic email-sending,  its own group, AlwaysCommit
// Same handler, two topics, two groups → two independent lanes in one pod.

The routing side is TopicByPriority(resolvePriority(clientID)) (.../grpc/priority.go:11): a client marked high-priority publishes to the .high-priority topic; everyone else to the normal one.

Same pattern, twice notification uses the identical shape for notification.async-notification-upserting and its .high-priority sibling — and conversationmgmt publishes urgent conversation pushes straight to notification's high-priority topic (infrastructure/kafka/notification.go:74). Once you see the pattern in spike, you recognise it across the team.

The trade-off (say this in the interview)

Two lanes buy isolation — urgent work is immune to a bulk backlog — at the cost of operational surface: more topics to create and monitor, and a producer that must know which lane to use. Because the lanes run as separate concurrent groups (not "always drain high first"), there's no starvation of the normal lane — both make progress at once. That's the distinction between this and a naive priority queue, and it's exactly the nuance an interviewer is listening for.

Confluent — Implementing Message Prioritization in Apache Kafka

Confirms Kafka has no native priority and walks through the separate-topics approach (what we use) plus the "bucket priority" alternative. See also KIP-349 for why it isn't built in.

→ confluent.io/blog/prioritize-messages-in-kafka
→ KIP-349 — Priorities for Source Topics

Check yourself (from memory)

Q1. Why can't a "priority" field in the payload make urgent messages go first?

Order is fixed by offset within a partition; the consumer receives events in log order regardless of any field's value.

Q2. Our two-lane pattern isolates urgent work by using…

Topic-per-priority + group-per-topic, running concurrently. A backlog on one lane can't touch the other — different queue entirely.

Why doesn't the normal lane starve, if the high lane is drained "first"?

recall, then click to reveal

Because the two lanes aren't "high-first then low" — they're two independent consumer groups on two topics, each in its own goroutine, running concurrently. Both make progress at the same time; "start high first" is just startup order. Isolation without starvation.

Want to compare our separate-topics approach with Confluent's "bucket priority" (partitions-as-buckets) pattern, and when you'd pick each? Ask me.

1. Confluent — Implementing Message Prioritization in Apache Kafka; repo: repo-kafka-map.md (lanes).