Checkpoint — and the 20 GB WAL
Checkpoints copy frames back to the DB file and rewind the WAL — but a long-lived reader can pin the WAL open forever, blowing it up.
Every commit appends to the WAL and the DB file 'catches up later.' Checkpointing is what 'later' means — and it has one famous failure mode.
Scene 09
Checkpoint — and the 20 GB WAL
- Watch
- Try it
- Predict
- Capture
Auto-checkpoint fires when the WAL hits its threshold. Watch the checkpoint pointer walk every frame, copy it into the matching DB page (the strip lights up), and — because no reader is pinned — the WAL truncates to zero.
Highlighted lines are the ones running in the diagram right now.
def checkpoint_passive(wal, db, readers):floor = min((r.snapshot_end for r in readers), default=None)for frame in wal.frames:db.write_page(frame.page_number, frame.bytes)if floor is None:wal.rewind_to_start() # next commit overwrites from f1# else: cannot rewind past floor; WAL keeps growingreturn
def checkpoint_truncate(wal, db, readers):wait_for_writers_to_finish() # FULL semanticsfor frame in wal.frames: # transfer EVERY framedb.write_page(frame.page_number, frame.bytes)if any(r.snapshot_end is not None for r in readers):return # pinned reader → cannot rewind, file stays fullwal.rewind_to_start()wal.truncate_file_to_zero_bytes() # physical shrink
def on_commit(wal, shm, threshold = 1000):wal.append_commit_frame() # page image + commit markerif synchronous in (FULL, EXTRA):wal.fsync() # durability pointshm.publish_new_end_mark(len(wal.frames))if len(wal.frames) >= threshold:# auto-checkpoint runs PASSIVE on this connectioncheckpoint_passive(wal, db, readers)
Where this sits in Build a B-tree storage engine (SQLite-style)
Scene 09 of 11, in the Speed & durability act — Page cache, WAL+fsync, and the checkpoint that keeps WAL bounded.. Checkpoints copy frames back to the DB file and rewind the WAL — but a long-lived reader can pin the WAL open forever, blowing it up.
Up next. Now you can size, configure, and operate a SQLite database — workload by workload. The next scene is the design canvas where you put it all together.
All 11 scenes in Build a B-tree storage engine (SQLite-style) · Every curriculum