A chunk: 120 points, packed

A CHUNK (the new term) is a fixed-size bundle of consecutive points from one series — bit-packed into one blob with one header — and Prometheus closes it at 120 samples or 2 hours, whichever comes first, because that is where Gorilla's 1.37 B/point compression averages out (vs 16 B/point naive: a ~12× win).

Previously

We have two encodings, each almost magical alone. A chunk is where they live together.

Scene 05

A chunk: 120 points, packed

Watch
Try it
Predict
Capture

What to watch for

New term this scene: CHUNK — a fixed-size bundle of compressed points from one series. Watch 120 samples accumulate, the bitstring pack in lockstep, and the chunk close. Then the bars draw and the 12× ratio appears.

Implementation

Highlighted lines are the ones running in the diagram right now.

Chunk

one series, one header, one bit-packed body

1class Chunk:
2    # --- header (16 B, written once) ---
3    series_id:   uint64       # which series
4    first_ts:    int64        # anchor for delta-of-delta
5    first_value: float64      # anchor for XOR
6    # --- body (bit-packed, grows append-only) ---
7    bits:        BitBuffer    # (ts_bits, val_bits) pairs
8    samples:     int = 0      # cap at SAMPLES_CAP (120)
9    span_min:    int = 0      # cap at TIME_CAP_MIN (120)
10    closed:      bool = False # immutable once true

Chunk.append

encode dod + xor; close on whichever cap fires first

1def append(chunk, ts, value):
2    dod = (ts - chunk.last_ts) - chunk.last_delta
3    chunk.bits.write(encode_dod(dod))   # ~1 bit on cadence
4    xor = float_bits(value) ^ chunk.last_value_bits
5    chunk.bits.write(encode_xor(xor))   # 1 bit if unchanged
6    chunk.samples  += 1
7    chunk.span_min  = (ts - chunk.first_ts) // 60
8    if chunk.samples  >= SAMPLES_CAP \
9       or chunk.span_min >= TIME_CAP_MIN:
10        close_chunk(chunk)

close_chunk

finalize, mark immutable, hand to mmap

1def close_chunk(chunk):
2    chunk.bits.flush_byte_aligned()  # body ends on byte boundary
3    chunk.closed = True              # no more appends, ever
4    # one chunk = the unit of compression / mmap / I/O
5    file = chunks_dir / f"{chunk.series_id}-{chunk.first_ts}"
6    file.write(chunk.header || chunk.bits)
7    mmap_readonly(file)              # query path reads via mmap
8    # head opens a fresh chunk for this series and keeps scraping
9    return Chunk(series_id=chunk.series_id)

Where this sits in Build a Prometheus-style time-series database

Scene 05 of 12. Bundle ~120 consecutive points into a single bit-packed blob. Gorilla: 16 B/point → 1.37 B/point — about 12× compression.

Up next. A chunk is great when it's full. But points arrive one at a time — where does a not-yet-full chunk live, and what happens if the process crashes mid-chunk?

All 12 scenes in Build a Prometheus-style time-series database · Every curriculum

Built with Arqly

Every scene in Build a Prometheus-style time-series database builds on the one before it.All 12 Build a Prometheus-style time-series database scenes