Lifecycle and tiering: cheap because bytes are immutable

Build an S3-style distributed object store (12 scenes)

Scene 10 · Lifecycle and tiering: cheap because bytes are immutable

Storage classes trade retrieval latency for cost; a lifecycle rule slides an object down the ladder as a pointer move.

Previously

Reaping orphaned parts was one declarative lifecycle action — and the same lifecycle machinery drives the bigger cost lever: sliding objects to cheaper storage as they cool.

Scene 10

Lifecycle and tiering: cheap because bytes are immutable

Watch

Diagram

A vertical ladder of storage classes from hot Standard (ms, highest $/GB, no minimum) down to Glacier Deep Archive (hours, lowest $/GB, 180-day minimum). The green object tile sits on whichever rung the slider picks; a lifecycle-rule card is read by a background worker that slides the object down — only the index pointer follows, the bytes are rewritten off-stage, and the key never changes.

Sources

Storage class = a tier with its own per-GB cost, retrieval latency, and minimum-duration lock-in.

The object starts on Standard — the hot rung: ms reads, highest per-GB cost, no minimum-duration commitment. A lifecycle rule says "transition after 30 days." Watch the day counter: when it fires, a background worker rewrites the bytes one rung colder and the index pointer follows. The key never changes; no client sees the move — but the retrieval-latency readout climbs.

Implementation

LifecycleWorker.scan

background fleet sweeps the prefix and fires due rules

1# runs continuously over each bucket/prefix
2for obj in index.scan(prefix):
3    age = now() - obj.createdAt
4    if age >= rule.transitionAfterDays:
5        transition(obj, rule.toClass)  # slide one rung colder
6    if age >= rule.expireAfterDays:
7        expire(obj)                    # delete current version
8# orphaned multipart parts never showed up in LIST
9for mpu in index.incompleteUploads(prefix):
10    if mpu.age >= rule.reapIncompleteMpuAfterDays:
11        abortUpload(mpu)               # parts leave billing meter

Worker.transition

the move is a byte rewrite plus an index-pointer flip

1def transition(obj, toClass):
2    # immutable bytes => no in-place edit to coordinate
3    newLoc = dataPlane.rewrite(
4        obj.fragments, toClass.medium,
5    )
6    # index is the source of truth for location + class
7    index.update(obj.key, {
8        location: newLoc,
9        storageClass: toClass,
10    })  # key unchanged; no client sees the move
11    dataPlane.free(obj.oldLocation)

Billing.onDelete

the cold rate is bought with a minimum-duration lock-in

1def onDelete(obj):
2    stored = now() - obj.classEnteredAt
3    minDays = obj.storageClass.minDurationDays
4    # cold classes bill the floor even if you delete early
5    billedDays = max(stored, minDays)
6    charge(obj, billedDays)

Not sure what to ask? Tap a question — the staff engineer answers in the chat panel.

PreviousMultipart upload and the ETag-that-isn't-an-MD5 NextDesign canvas: defend your durability number