Build a columnar OLAP store (ClickHouse / Druid style) (13 scenes)
Scene 02

Same table, two on-disk shapes — row-store pages versus per-column .bin files

A row store interleaves all columns of one row contiguously on a page; a column store stores all values of one column contiguously in its own file — same rows, rotated 90 degrees.

Previously

If both engines hold the same 1.2 billion rows, the only thing that can explain a 9000x gap is what each engine actually pulls off disk — so we need to look at the physical layout under the rows. Here's that physical layout: same rows, two arrangements.

Scene 02

Same table, two on-disk shapes

  1. Watch
  2. Try it
  3. Predict
  4. Capture
idcountrylatency_msts1US4217000000002US5117000000013DE3917000000024US4417000000035DE601700000004Row store on diskone big strip: [row1][row2]…cursor pays full row width: reads all 4 tiles × 5 rows = 160 B (uses 1)Column store on disk4 files, one per columnid.bin12345country.binUSUSDEUSDElatency_ms.bin4251394460ts.bin17000000001700000001170000000217000000031700000004streams: opens 1 of 4 files = 40 BSELECT avg(latency_ms) — Narrow query: row store sweeps every cell; column store streams one strip (latency.bin) end-to-end with …
page (row store): one file, every column on every row →
↑ row store reads all 4 tiles per row even though the query needs 1
column file: one .bin per column — only latency_ms.bin glows →
What to watch for

Same five rows, two layouts. The row strip is the page on disk — every column of every row glued together. The four .bin strips are the column files — each one holds one column end-to-end. SELECT avg(latency_ms) lights latency_ms in both panels: the row store sweeps every tile anyway; the column store streams exactly one file.

Implementation

Highlighted lines are the ones running in the diagram right now.

RowStore.scan(query)
open the page, walk every row, skip past unwanted columns
def scan(query):
    page = openPage('events.page')
    out = []
    for row_offset in page.row_offsets:  # every row
        row = []
        for col in SCHEMA:  # all 4 columns, every time
            bytes = page.read(row_offset, col.width)
            if col.name in query.columns:
                row.append(decode(bytes, col.type))
            row_offset += col.width  # skip past unwanted
        out.append(row)
    return out
ColumnStore.scan(query)
open one .bin per column, stream end-to-end, then zip
def scan(query):
    streams = [open(f'{c}.bin') for c in query.columns]
    if len(streams) == 1:
        return list(streams[0])  # zero-waste single stream
    return zipByOrdinal(streams)  # tuple reconstruction
ColumnStore.zipByOrdinal(streams)
tuple reconstruction — position k in each file is row k
def zipByOrdinal(streams):
    out = []
    for k in range(rowCountInPart()):
        # value at position k in each .bin belongs to row k.
        # no row_id stored; alignment is purely by index.
        row = tuple(stream.readAt(k) for stream in streams)
        out.append(row)
    return out
Sources

Where this sits in Build a columnar OLAP store (ClickHouse / Druid style)

Scene 02 of 13, in the Why columnar? act — The 30-min Postgres query vs the 200ms ClickHouse query — what's on disk?. Row store interleaves a row's columns contiguously; column store stores each column in its own file. Same rows, rotated 90°.

Up next. Once adjacent bytes on disk are the same column — same type, often similar values — compression that fails on row pages suddenly works. Let's see how much.

All 13 scenes in Build a columnar OLAP store (ClickHouse / Druid style) · Every curriculum

Built with Arqly
Every scene in Build a columnar OLAP store (ClickHouse / Druid style) builds on the one before it.All 13 Build a columnar OLAP store (ClickHouse / Druid style) scenes