Compression — the column store's superpower

When adjacent bytes are the same column they are the same type and often the same or similar values, so run-length encoding and dictionary encoding deliver 5-20x compression that fails completely on row pages — and the encoded form is what the executor reads.

Previously

Once adjacent bytes are the same column, compression that fails on row pages suddenly works — and the win is per-column, not generic gzip.

Scene 03

Compression: the column store's superpower

Watch
Try it
Predict
Capture

What to watch for

country.bin streams in — 20 rows of a low-cardinality string column (US, DE, JP). The byte counter top-right tracks the cost of the raw layout: 2 bytes per row, 40 bytes total. Notice how adjacent rows often share the same value — that's the property the next mode will exploit.

Implementation

Highlighted lines are the ones running in the diagram right now.

Column.rleEncode(values)

collapse adjacent repeats into (value, run_length) tuples

1def rle_encode(values):
2    runs = []
3    for v in values:
4        last = runs[-1] if runs else None
5        if last and last.value == v:
6            last.run += 1   # extend current run
7        else:
8            runs.append(Run(value=v, run=1))
9    return runs            # on-disk form for RLE columns

Executor.countWhere(column, value='US')

the same WHERE clause, three encodings, three scans

1def count_where(col, target):
2    if col.mode == RAW:
3        return sum(1 for v in col.values if v == target)
4    if col.mode == RLE:
5        # no decompression — sum run lengths of matching runs
6        return sum(r.run for r in col.runs if r.value == target)
7    if col.mode == DICT:
8        code = col.dictionary.codeFor(target)  # 'US' -> 2
9        return sum(1 for c in col.codes if c == code)

Column.dictEncode(values)

intern each distinct value; column becomes integer codes

1def dict_encode(values):
2    dictionary = {}        # value -> small int code
3    codes = []
4    for v in values:
5        if v not in dictionary:
6            dictionary[v] = len(dictionary)  # next code
7        codes.append(dictionary[v])
8    # cardinality check — the LowCardinality(String) trap
9    # fires when len(dictionary) approaches len(values).
10    return codes, dictionary

Where this sits in Build a columnar OLAP store (ClickHouse / Druid style)

Scene 03 of 13, in the Speedups act — Compression and vectorized execution — where the orders of magnitude live.. Adjacent column values are same-type and often similar, so RLE and dictionary encoding deliver 5–20× shrinkage that fails completely on row pages.

Up next. The bytes on disk are now tiny. But pulling tiny bytes through the CPU one row at a time still leaves a 50x performance win on the table — it depends on how the executor walks those bytes.

All 13 scenes in Build a columnar OLAP store (ClickHouse / Druid style) · Every curriculum

Built with Arqly

Every scene in Build a columnar OLAP store (ClickHouse / Druid style) builds on the one before it.All 13 Build a columnar OLAP store (ClickHouse / Druid style) scenes