Arqly
Sign in
#28Build a columnar OLAP store (ClickHouse / Druid style)
stub
OLTP picks one row by key; OLAP scans a billion rows of one column and asks for a percentile. Build the analytical engine that makes that fast: columnar layout, dictionary/RLE/delta compression, vectorized execution, late materialization, MPP shuffle. Internalize why Postgres is 1000× slower than ClickHouse on the same query and why the inverse is also true.

Build a columnar OLAP store (ClickHouse / Druid style). OLTP picks one row by key; OLAP scans a billion rows of one column and asks for a percentile. Build the analytical engine that makes that fast: columnar layout, dictionary/RLE/delta compression, vectorized execution, late materialization, MPP shuffle. Internalize why Postgres is 1000× slower than ClickHouse on the same query and why the inverse is also true.

This problem is a stub. Suggested approaches haven't been authored yet — ask the staff engineer in the right panel for any stage and they'll generate one tuned to your draft.
Builds on: LSM trees — 2-min primers appear where needed.
Reading: Stonebraker et al. — C-Store: A Column-oriented DBMS (VLDB 2005) · Abadi et al. — Column-Stores vs. Row-Stores: How Different Are They Really? (SIGMOD 2008) · ClickHouse documentation — Architecture, MergeTree, query pipeline · Apache Druid — Design paper (Druid: A Real-time Analytical Data Store, SIGMOD 2014) · Snowflake — The Snowflake Elastic Data Warehouse (SIGMOD 2016) · Boncz et al. — MonetDB/X100: Hyper-Pipelining Query Execution (CIDR 2005) · Apache Arrow — In-memory columnar format spec
row-store vs column-store (the load-bearing layout choice)
dictionary + RLE + delta + LZ4 compression by column type
vectorized execution — process one column-chunk at a time, not one row
predicate pushdown + zone maps / min-max indexes
late materialization — defer row reconstruction to after filters
MergeTree / LSM-on-columns: parts, merges, mutations
MPP query execution: scan → shuffle → aggregate → final
sharding by primary key vs by date partition
approximate queries: HyperLogLog, T-Digest, quantile sketches
ETL pull (batch) vs streaming ingest with idempotent inserts