Lesson 8 · Internals & data structures

WAL: the write-ahead log

Durability, crash recovery — and the log that literally feeds your Kafka events.

Your win: explain what the WAL is, what it guarantees, and why it's fast — plus the concrete tie between a Postgres commit and a Kafka message in this repo.

Write the log before the data

A crash mid-write could leave data files half-updated. Postgres prevents that with the write-ahead log: before changing a data page, it appends a record describing the change to the WAL. The rule — log first, then data — is where the name comes from.1

1. change happens in memory (a page in the buffer cache) 2. WAL record appended + flushed to disk ← on COMMIT (fsync) 3. the data page itself is written later, lazily (checkpoint) crash after step 2? → on restart, REPLAY the WAL to rebuild the data pages.
What it buys: the "D" in ACID Once a transaction's WAL records are flushed at COMMIT, the change is durable — it survives a crash, because recovery replays the WAL. The data files can lag behind safely. The WAL is also the basis of replication (stream it to a replica) and point-in-time recovery.

Why it's fast

The WAL is a sequential append — the disk head (or SSD) just writes forward, far cheaper than scattering random in-place writes across data files at commit time. Batching many changes into one sequential log is the trick.

You already know this shape "Append to a durable log first, apply later" is exactly the Kafka insight from your Kafka course. Sequential append = durability + speed, whether it's Postgres's WAL or Kafka's partition log. One idea, two systems.
Anchor — the tie is literal here Your ExecInTx COMMIT is durable precisely because its WAL was flushed. And in the local stack, Postgres runs logical replication off the WAL to feed Debezium → Kafka CDC events (see the repo map). So a committed row change in emails becomes a Kafka message via the WAL — your Postgres and Kafka courses meeting in one pipeline.
Read this next

PostgreSQL — Write-Ahead Logging (WAL) + Rogov Internals Part II

The reliability chapter on why WAL exists, then Rogov's deep treatment of WAL & the buffer cache.

→ postgresql.org/docs — WAL intro
→ PostgreSQL 14 Internals — Part II (Buffer Cache & WAL)

Check yourself (from memory)

Q1. The WAL is written…

Log-first: the change is logged (and flushed on commit) before the data page is written. That ordering is the whole guarantee.

Q2. The WAL primarily provides…

The "D" in ACID — committed changes survive a crash via replay. Also underpins replication & PITR.

Q3. Writing the WAL is fast because it is…

Sequential appends beat scattered random writes — the same reason Kafka's log is fast.
What is the WAL, what does it guarantee, and how does it connect to Kafka here?
recall, then click to reveal
The Write-Ahead Log is an append-only log of every change, written and flushed to disk BEFORE the corresponding data pages are modified. It guarantees DURABILITY (the D in ACID): once a COMMIT's WAL is flushed, the change survives a crash — recovery replays the WAL. Sequential append makes it fast (the same insight as Kafka's log). In this repo the tie is literal: the local stack runs logical replication off the WAL to feed Debezium → Kafka CDC, so a committed Postgres write becomes a Kafka message via the WAL.
Want to understand checkpoints, synchronous_commit, or how streaming replication rides the WAL? Ask me.

1. PostgreSQL — Write-Ahead Logging.