Segments — many small immutable indexes

Lucene never mutates an inverted index in place: each new write lands in an in-memory buffer that is periodically sealed into a brand-new immutable SEGMENT (the new term — a small self-contained inverted index), deletes are recorded as a flipped bit in a per-segment .liv bitset rather than a rewrite, and a background merge controller asynchronously combines small segments into larger ones — so reads stay lock-free and the cost of mutability is paid in the background.

Previously

The inverted index is beautiful when frozen and miserable when mutated. Every PUT would touch the term dictionary at random terms; every DELETE would shift posting lists. The fix is to never mutate.

Scene 03

Segments — many small immutable indexes

Watch
Try it
Predict
Capture

What to watch for

New term this scene: SEGMENT — a small, immutable, self-contained inverted index. Watch the five canonical books arrive: books 1-3 fill the in-memory buffer and seal into seg_001, then books 4-5 arrive into a fresh buffer and seal into seg_002, then a merge controller fuses both into one larger segment. seg_001 and seg_002 are never edited — they fade out and a brand-new segment takes their place.

Implementation

Highlighted lines are the ones running in the diagram right now.

indexer.seal_buffer

buffer fills → freeze → emit a new immutable segment

1def index(doc):
2    buffer.append(doc)            # in-memory only
3    if buffer.full():
4        seg = open_new_segment(next_id())
5        for d in buffer.iter():
6            seg.add_to_inverted_index(d)
7        seg.seal()                # immutable forever
8        live_segments.append(seg) # readers pick it up on next snapshot
9        buffer = new Buffer()     # fresh tray for the next batch

indexer.delete

delete = flip a .liv bit; the segment is never rewritten

1def delete(doc_id):
2    seg = segment_holding(doc_id)
3    seg.liv.clear_bit(doc_id)     # one bit, out-of-band
4    # the inverted-index posting list is UNCHANGED.
5    # search() will filter doc_id out via seg.liv on read.
6    # bytes are reclaimed only when seg is merged.

merge_controller.tick

background: pick small segments, emit one bigger one, atomic swap

1def maybe_merge(live_segments):
2    candidates = pick_small_adjacent(live_segments)
3    if not candidates: return
4    out = open_new_segment(next_id())
5    for seg in candidates:
6        for doc in seg.iter_live_docs():   # skips .liv-cleared docs
7            out.add_to_inverted_index(doc)
8    out.seal()
9    atomic_swap(remove=candidates, add=[out])  # readers never see a mix

Where this sits in Build a distributed search engine (Elasticsearch / OpenSearch style)

Scene 03 of 12. Lucene appends a fresh tiny inverted index per write batch and merges in the background — readers never lock, deletes are bit flips, mutability is an asynchronous receipt.

Up next. We've quietly assumed the buffer-seal is instant and free. It isn't — and the segments don't reach disk for free either. If a segment is sealed only when it's durable on disk, brand-new writes are invisible to search until the next disk sync. The next scene splits 'sealed and searchable' from 'sealed and durable' and shows what protects the gap between them.

All 12 scenes in Build a distributed search engine (Elasticsearch / OpenSearch style) · Every curriculum

Designs that use this

Built with Arqly

Every scene in Build a distributed search engine (Elasticsearch / OpenSearch style) builds on the one before it.All 12 Build a distributed search engine (Elasticsearch / OpenSearch style) scenes