Checkpoint — and the 20 GB WAL

Checkpoints copy frames back to the DB file and rewind the WAL — but a long-lived reader can pin the WAL open forever, blowing it up.

Previously

Every commit appends to the WAL and the DB file 'catches up later.' Checkpointing is what 'later' means — and it has one famous failure mode.

Scene 09

Checkpoint — and the 20 GB WAL

  1. Watch
  2. Try it
  3. Predict
  4. Capture
users.db — pages catch up as checkpoint flushes123456789101112131415161718192021222324users.db-wal — 8 frames, 32 KBf1p2f2p7f3p12f4p17f5p22f6p5f7p10f8p15users.db-shmrewind possible: notruncating: noflushed ≠ rewoundSCENARIOhealthylong readerTRUNCATE okTRUNCATE stuckWAL stays small · 0 checkpoints run · auto-truncate worksWAL SIZE NOW32 KBpeak 32 KBCHECKPOINTS0WAL SIZE TRAJECTORYno samples yetcheckpoint = copy frames home + rewind the WAL to zero
What to watch for

Auto-checkpoint fires when the WAL hits its threshold. Watch the checkpoint pointer walk every frame, copy it into the matching DB page (the strip lights up), and — because no reader is pinned — the WAL truncates to zero.

Continue unlocks when the animation finishes.
Implementation

Highlighted lines are the ones running in the diagram right now.

Checkpoint.passive
default mode — copy what you can, never block writers
def checkpoint_passive(wal, db, readers):
floor = min((r.snapshot_end for r in readers), default=None)
for frame in wal.frames:
db.write_page(frame.page_number, frame.bytes)
if floor is None:
wal.rewind_to_start() # next commit overwrites from f1
# else: cannot rewind past floor; WAL keeps growing
return
Checkpoint.truncate
FULL-equivalent flush, then physically zero the WAL file
def checkpoint_truncate(wal, db, readers):
wait_for_writers_to_finish() # FULL semantics
for frame in wal.frames: # transfer EVERY frame
db.write_page(frame.page_number, frame.bytes)
if any(r.snapshot_end is not None for r in readers):
return # pinned reader → cannot rewind, file stays full
wal.rewind_to_start()
wal.truncate_file_to_zero_bytes() # physical shrink
Pager.onCommit
what each commit does — and what fires the checkpoint
def on_commit(wal, shm, threshold = 1000):
wal.append_commit_frame() # page image + commit marker
if synchronous in (FULL, EXTRA):
wal.fsync() # durability point
shm.publish_new_end_mark(len(wal.frames))
if len(wal.frames) >= threshold:
# auto-checkpoint runs PASSIVE on this connection
checkpoint_passive(wal, db, readers)

Where this sits in Build a B-tree storage engine (SQLite-style)

Scene 09 of 11, in the Speed & durability act — Page cache, WAL+fsync, and the checkpoint that keeps WAL bounded.. Checkpoints copy frames back to the DB file and rewind the WAL — but a long-lived reader can pin the WAL open forever, blowing it up.

Up next. Now you can size, configure, and operate a SQLite database — workload by workload. The next scene is the design canvas where you put it all together.

All 11 scenes in Build a B-tree storage engine (SQLite-style) · Every curriculum

Built with Arqly
Every scene in Build a B-tree storage engine (SQLite-style) builds on the one before it.All 11 Build a B-tree storage engine (SQLite-style) scenes