Append, fsync, update, ack

Build a Bitcask-style KV store (9 scenes)

Scene 02 · Append, fsync, update, ack

Every put is four ordered steps on one open writer. Concurrent writers are rejected at open(), not at write time.

Previously

The split is the shape; the order is the contract. Append, then fsync, then keydir update, then ack — and one open writer at a time. That ordering is what makes the disk and the RAM agree.

Scene 02

Append, fsync, update, ack

Watch

Diagram

Segments left, keydir right; the active tail lights 1 append → 2 fsync → 3 keydir → 4 ack. A rejected second writer and an ops/sec meter sit beside it.

Sources

A single producer issues PUT k1=v1. Watch the four steps light up in order — append at the tail, fsync, keydir row updated, ack to the client — and the throughput meter climb as more puts stream in.

Implementation

Bitcask.put_record

the four ordered steps every put goes through

1def put_record(key, value):
2    rec = encode(crc, tstamp, ksz, vsz, key, value)
3    # 1) append at end-of-file on the active file
4    pos = active_file.size
5    active_file.write(rec)        # sequential append
6    # 2) fsync per sync_strategy (none | o_sync | interval)
7    if sync_strategy == 'o_sync':
8        active_file.fsync()
9    # 3) update keydir entry in place
10    keydir[key] = (active_file.id, vsz, pos, tstamp)
11    # 4) ack to the client
12    return ok

Bitcask.open_for_write

the writer lock is acquired once, at open() — not per write

1def open(dir, mode):
2    if mode == read_write:
3        # one writer per directory, enforced here
4        lock = try_flock(dir + '/bitcask.write.lock')
5        if lock is None:
6            return error('already open for writing')
7        active_file = open_or_create_active(dir)
8        keydir = scan_or_load_hint_files(dir)
9        return Handle(dir, active_file, keydir, lock)
10    # read-only handles share freely
11    return ReadOnlyHandle(dir)

Bitcask.maybe_rotate

active file crosses max_file_size: close it, open the next one

1def maybe_rotate(active_file):
2    if active_file.size < max_file_size:  # default 2 GB
3        return active_file
4    # 1) close the current active file — now immutable
5    active_file.close()
6    immutable_files.append(active_file)
7    # 2) open a fresh active file for subsequent appends
8    next_id = active_file.id + 1
9    return create_active(dir, next_id)
10    # rotation is not merge: closed bytes are unchanged here

PreviousLog on disk, hash in RAM NextOne hash lookup, one pread