#15Build a CDC pipeline (Debezium + outbox)
Your service writes to its DB and publishes to Kafka — and any crash between those two writes is permanent inconsistency. Build a Change Data Capture pipeline (modeled on Debezium + the outbox pattern) that closes the gap by making the database itself the event source.

You have a service. It writes to its database and it publishes to a queue — and you have been bitten (or nearly bitten) by the gap between those two writes. The database commit succeeded but the queue publish failed; the publish ack got lost so a retry duplicated the event; two writers raced and the queue's order disagreed with the database's last-writer-wins. Each is a permanent inconsistency: a state the system cannot heal on its own, because the system is two systems with no shared atomicity boundary.

This curriculum is about closing that gap. The fix is Change Data Capture — turning the database's own write log into a stream of change events — and the engineering pattern that makes it land in production: the outbox pattern, where every business write also writes a row whose entire purpose is to be published. Modeled on Debezium and Postgres logical replication.

You start from the user's mental model. "I have a service that writes to its DB. I want other services to know about my changes without breaking my service." From that one frame, every concept arrives motivated: the WAL exists because the database needs it for recovery; the replication slot exists because the database needs to know how far the consumer has read; the snapshot exists because the WAL doesn't go back forever; the outbox exists because the service still wants atomicity. Eleven scenes plus a capstone, no slideshow.

The single insight that unlocks the design: the database's own write log is the source of truth your downstream consumers wanted all along — the outbox makes that source carry your domain events instead of your row deltas.

Builds on: Kafka — 2-min primers appear where needed.
Reading: Debezium documentation — debezium.io/documentation/ · Martin Kleppmann — Online Event Processing (Queue, 2019) · Gunnar Morling — Reliable Microservices Data Exchange With the Outbox Pattern (debezium.io/blog) · Designing Data-Intensive Applications, Ch 11 — Stream Processing · Postgres docs — Logical replication + replication slots · MySQL docs — Binary log overview
the dual-write problem
polling CDC vs log-based CDC
WAL / binlog as the event source
Debezium connector + change events
replication slot + LSN
consistent snapshot, then stream
schema registry + compatibility modes
outbox pattern (atomic event publish)
outbox cleanup (tombstone + log compaction, partition-drop, INSERT+DELETE same-tx)
partition-by-aggregate ordering
at-least-once + idempotent consumer