Multipart upload and the ETag-that-isn't-an-MD5

A multipart object's ETag is a hash of the part hashes with a "-N" suffix, NOT the MD5 of the file — so the same bytes uploaded with a different part size produce a different ETag.

Previously

With the index telling the truth and the object stored durably, the last piece of an object's life is the upload itself — and objects too big for one request expose a famous integrity gotcha.

Scene 09

Multipart upload and the ETag-that-isn't-an-MD5

Watch
Try it
Predict
Capture

What to watch for

A 200 MiB file is sliced into 4 parts. Each uploads in parallel along its own arrow and returns its own ETag into the manifest. Once all four land, the server runs CompleteMultipartUpload and stitches them into one object — whose ETag renders as a hash-of-hashes ending in "-4".

A large file on the left is sliced into N parts (each ≥ 5 MiB except the last) that upload in parallel; each returns its own ETag.
The MANIFEST collects the (partNumber, ETag) pairs, and on completion the server stitches the parts into one object whose ETag is a hash-of-hashes ending in "-N".
This whole split-upload-stitch flow is a MULTIPART UPLOAD; the trailing "-N" is the tell that the ETag is md5(concatenated part md5s), not md5(file).

Implementation

Highlighted lines are the ones running in the diagram right now.

Client.multipartUpload

slice the file, upload parts in parallel, then complete

1def multipartUpload(file, partSize):
2    uploadId = s3.createMultipartUpload(key)
3    manifest = []
4    for n, chunk in enumerate(slice(file, partSize), 1):
5        # each part >= 5 MiB except the last, partNumber 1..10000
6        etag = s3.uploadPart(uploadId, n, chunk)
7        manifest.append((n, etag))   # (partNumber, ETag)
8    return s3.completeMultipartUpload(uploadId, manifest)

Server.completeMultipartUpload

stitch the parts and compute the object ETag

1def completeMultipartUpload(uploadId, manifest):
2    parts = sortByPartNumber(manifest)   # order, not arrival
3    stitchIntoObject(parts)
4    if len(parts) == 1 and wasSinglePut(uploadId):
5        objectEtag = md5(object.bytes)        # == md5(file)
6    else:
7        joined = concat(md5Binary(p) for p in parts)
8        objectEtag = md5(joined) + "-" + str(len(parts))
9    return objectEtag

Sources

Where this sits in Build an S3-style distributed object store

Scene 09 of 12, in the Make it real act — Strong consistency, multipart upload, lifecycle & tiering.. Parallel resumable parts, a completion manifest, and why the object ETag is a hash-of-hashes ending in -N.

Up next. That orphaned-parts cleanup rule is one declarative lifecycle action — and lifecycle rules also drive the bigger cost lever: sliding objects down to cheaper storage as they cool.

All 12 scenes in Build an S3-style distributed object store · Every curriculum

Built with Arqly

Every scene in Build an S3-style distributed object store builds on the one before it.All 12 Build an S3-style distributed object store scenes