Refresh, flush, translog — three cadences

Refresh makes a write searchable, flush makes it durable on disk, and the translog is the write-ahead log that bridges the two — three cadences, three independent properties, one famous source of confusion.

Previously

Sealing a segment to disk costs hundreds of milliseconds; real users want sub-second visibility AND a guarantee they won't lose acknowledged writes. Three clocks let visibility and durability run on different cadences, with a write-ahead log bridging the gap.

Scene 04

Refresh, flush, translog — three cadences

Watch
Try it
Predict
Capture

What to watch for

A new book is indexed. Step 1: it lands in the IndexWriter buffer and the translog. Step 2: at the next refresh tick, the buffer becomes a new searchable segment. Step 3: at flush, the segment is committed to disk and the translog is truncated.

Implementation

Highlighted lines are the ones running in the diagram right now.

index(doc)

append to translog, append to in-memory buffer; ack only after fsync

1def index(doc):
2    buffer.append(doc)                  # in-memory, not searchable yet
3    translog.append(op(doc))            # WAL row
4    if durability == 'request':
5        translog.fsync()                # acked write is durable
6    return ack

refresh()

seal the buffer into a fresh in-memory segment — visibility, not durability

1def refresh():                          # every refresh_interval
2    if buffer.empty(): return
3    seg = open_new_in_memory_segment()
4    for doc in buffer.drain():
5        seg.add_to_inverted_index(doc)
6    seg.seal()                          # searchable now
7    live_segments.append(seg)           # NOT on disk yet

flush()

IndexWriter.commit() — segments_N advances, translog truncates

1def flush():                            # every flush_threshold
2    refresh()                           # drain any pending buffer
3    index_writer.commit()               # fsync segments_N
4    translog.rotate_and_truncate()      # WAL no longer needed

Where this sits in Build a distributed search engine (Elasticsearch / OpenSearch style)

Scene 04 of 12. Refresh = visible to search; flush = survives a crash; translog bridges the gap. Three cadences, three durability properties, one famous source of confusion.

Up next. Everything so far fits on one machine. Five million books at 2 KB of body each is 10 GB of postings — fine for one node. Five hundred million books is not, and the routing decision has to be made before the document even reaches an inverted index.

All 12 scenes in Build a distributed search engine (Elasticsearch / OpenSearch style) · Every curriculum

Designs that use this

Built with Arqly

Every scene in Build a distributed search engine (Elasticsearch / OpenSearch style) builds on the one before it.All 12 Build a distributed search engine (Elasticsearch / OpenSearch style) scenes