Why one row per point is wrong — per-row label repetition and 60-byte overhead

Storing each point as a SQL row spends 60+ bytes of metadata to carry an 8-byte float, because the label set is repeated on every row of an append-only firehose.

Previously

We have a clean four-part tuple — now we try to store it the obvious way and watch the bytes explode.

Scene 02

Why one row per point is wrong

Watch
Try it
Predict
Capture

What to watch for

Six scrapes of the same series stream in one at a time. Watch the labels JSON column (red) take more bytes than the row header, metric, ts, and value combined — every single row.

Implementation

Highlighted lines are the ones running in the diagram right now.

Storage.naiveInsert

one SQL row per point — labels JSON repeated every row

1def naiveInsert(point):
2    # rowHeader 8 + metric 22 + labels 60 + ts 8 + value 8
3    #   = ~106 B per row to carry an 8-byte float
4    db.execute(
5        'INSERT INTO points',
6        '  (metric, labels, ts, value)',
7        '  VALUES (?, ?, ?, ?)',
8        point.metric,        # 'http_requests_total'
9        json(point.labels),  # full label set, every row
10        point.ts,
11        point.value,
12    )
13

Storage.dedupeLabels

store the label set once; each row keeps a 2-byte ref

1labelDict = {}  # labelSet -> seriesId (16-bit)
2 
3def dedupeInsert(point):
4    key = canonical(point.labels)
5    if key not in labelDict:
6        labelDict[key] = nextSeriesId()
7    sid = labelDict[key]    # 2-byte ref
8    db.execute(
9        'INSERT INTO points (sid, ts, value)',
10        '  VALUES (?, ?, ?)',
11        sid, point.ts, point.value,
12    )
13    # row shrinks to ~28 B: header + 2B ref + ts + value

Storage.projectPerSeriesPerDay

multiply the per-row cost by the scrape count

1def projectStorage(
2    seriesCount, labelBytes, scrapesPerDay,
3):
4    rowHeader, metric, ts, value = 8, 22, 8, 8
5    perRow = rowHeader + metric + labelBytes + ts + value
6    perSeriesPerDay = perRow * scrapesPerDay
7    fleet = perSeriesPerDay * seriesCount
8    return perSeriesPerDay, fleet
9 
10# perRow is paid on every scrape — it doesn't amortize.
11# fleet scales the per-series number by seriesCount.

Where this sits in Build a Prometheus-style time-series database

Scene 02 of 12. Storing each point as a SQL row spends 60+ bytes of metadata to carry an 8-byte float — the labels JSON repeats on every row of the firehose.

Up next. The labels are the easy waste to fix — store them once. But timestamps are 8 bytes each forever, and there are billions of them. Can we shrink those too?

All 12 scenes in Build a Prometheus-style time-series database · Every curriculum

Built with Arqly

Every scene in Build a Prometheus-style time-series database builds on the one before it.All 12 Build a Prometheus-style time-series database scenes