Merge — and the 'too many parts' crash

A background worker continuously fuses small adjacent parts into bigger sorted parts; if the write rate exceeds the merge rate, the part count crosses parts_to_throw_insert and the engine rejects new INSERTs.

Previously

Many small parts pile up fast. If the count grows unbounded, every query has to interrogate every part — there must be a background process that fuses them, or the engine collapses.

Scene 07

Merge — and the 'too many parts' crash

Watch
Try it
Predict
Capture

What to watch for

Watch the steady-state baseline. Batched INSERTs land as new parts on the stack; the background merge worker picks 3-5 adjacent parts and fuses them into one bigger part at the next level. The active-parts counter hovers around 40 — well below the two thresholds.

Implementation

Highlighted lines are the ones running in the diagram right now.

MergeWorker.run

background loop: pick adjacent same-level parts, sort-merge, swap

1def merge_worker():
2    while True:
3        for partition in active_partitions():
4            # candidates = adjacent parts at the SAME level
5            picks = pick_adjacent_same_level(partition)
6            if len(picks) < 2:
7                continue
8            merged = sort_merge(picks)   # in sorted-key order
9            emit_larger_part(merged)     # level += 1
10            delete(picks)                # originals dim and go

MergeTree.insert_path

admission control: throttle at delay, reject at throw

1def insert_path(block, partition):
2    active_parts = count_active_parts(partition)
3    if active_parts > parts_to_throw_insert:   # default 3000
4        raise "Too many parts (" + str(active_parts) + ")"
5    if active_parts > parts_to_delay_insert:   # default 1000
6        sleep(backoff(active_parts))           # artificial throttle
7    new_part = write_part(block, partition)    # 1 INSERT = 1 part
8    register(new_part)

MergeTree.pick_adjacent_same_level

partition-key effect: merge is bounded by partition walls

1def pick_adjacent_same_level(partition):
2    parts = list_parts(partition)  # ONE partition only
3    # merges NEVER cross partition boundaries —
4    # a high-cardinality partition key (e.g. hourly)
5    # leaves each partition with too few adjacent
6    # same-level parts to fuse, so merge starves.
7    return longest_adjacent_run_at_same_level(parts)

Sources

Where this sits in Build a columnar OLAP store (ClickHouse / Druid style)

Scene 07 of 13, in the Write side act — Bulk inserts → immutable parts → background merge → too-many-parts cliff.. Background worker fuses small parts into bigger ones; when write rate exceeds merge rate, parts_to_throw_insert fires and inserts get rejected.

Up next. Merge keeps the part count bounded, but a query still has to descend into surviving parts. What it does inside each part — scan everything, or jump to a range — depends on how the rows were sorted when the part was written.

All 13 scenes in Build a columnar OLAP store (ClickHouse / Druid style) · Every curriculum

Built with Arqly

Every scene in Build a columnar OLAP store (ClickHouse / Druid style) builds on the one before it.All 13 Build a columnar OLAP store (ClickHouse / Druid style) scenes