Replication slot — the bookmark that fills disks — Build a CDC pipeline (Debezium + outbox)

Build a CDC pipeline (Debezium + outbox) (12 scenes)

Scene 05 · Replication slot — the bookmark that fills disks

A replication slot is a server-side cursor identified by an LSN that stops Postgres from recycling WAL the connector hasn't read — and an inactive slot is the #1 Debezium production failure.

Previously

Debezium tails the log and emits one change event per row change — but if it crashes, the database needs to know where it left off so the WAL can be safely recycled. Postgres provides exactly this bookmark; it's called a replication slot, and the position it remembers is an LSN.

Scene 05

Replication slot — the bookmark that fills disks

Watch

Diagram

**Replication slot** — the bookmark Postgres uses to remember how far the connector has read the WAL; while it exists, Postgres won't recycle WAL bytes the slot still needs. **LSN (Log Sequence Number)** — a monotonic byte offset into the WAL; identifies a position in the change stream. **WAL** — Postgres' append-only write-ahead log; every committed change becomes a WAL record. **Connector** — the Debezium process that subscribes to the slot and streams changes out as Kafka events.

Sources

Debezium needs to remember how far it's read so a restart doesn't lose its place. Postgres provides exactly this bookmark — a replication slot — pinned to an LSN. Watch restart_lsn slide forward in step with the write head while the connector is running. The disk stays green because Postgres can recycle WAL bytes behind the slot.

Implementation

Postgres.advanceRestartLsn(slot, newLsn)

Slot's restart_lsn moves only when the consumer acks.

1def advance_restart_lsn(slot, new_lsn):
2    if not slot.active:
3        # consumer gone -> restart_lsn frozen
4        return
5    if new_lsn <= slot.restart_lsn:
6        return  # never moves backward
7    slot.restart_lsn = new_lsn
8    # only NOW may pg_wal/ GC bytes < restart_lsn
9    wake_checkpointer()

Postgres.maybeRecycleWal(currentLsn)

GC refuses to recycle bytes any slot still claims.

1def maybe_recycle_wal(current_lsn):
2    pin = min(s.restart_lsn for s in pg_replication_slots)
3    for seg in pg_wal_segments():
4        if seg.end_lsn < pin:
5            recycle(seg)  # safe: no slot needs it
6        else:
7            keep(seg)     # slot pins this WAL
8    if disk_used(pg_wal) >= READONLY_THRESHOLD:
9        # cannot durably commit without WAL space
10        set_database_read_only()
11        refuse_further_writes()

Connector.confirmFlushedLsn()

Consumer tells Postgres how far it has durably read.

1def confirm_flushed_lsn():
2    while connector.running:
3        flushed = kafka_producer.last_acked_lsn()
4        # standby status update message
5        replication_stream.send_status(
6            write_lsn=flushed,
7            flush_lsn=flushed,
8            apply_lsn=flushed,
9        )
10        sleep(status_interval_ms)

PreviousDebezium tails the log NextSnapshot, then stream