Arqly
Sign in
#11Build an S3-style distributed object store
stub
Eleven nines of durability over disks that fail weekly. Build the object store from first principles — flat keyspace, immutable objects, erasure coding instead of replication, eventual consistency turned strong, multipart upload, lifecycle and tiering — and feel why every modern data lake sits on top of something shaped exactly like this.

Build an S3-style distributed object store. Eleven nines of durability over disks that fail weekly. Build the object store from first principles — flat keyspace, immutable objects, erasure coding instead of replication, eventual consistency turned strong, multipart upload, lifecycle and tiering — and feel why every modern data lake sits on top of something shaped exactly like this.

This problem is a stub. Suggested approaches haven't been authored yet — ask the staff engineer in the right panel for any stage and they'll generate one tuned to your draft.
Reading: DeCandia et al. — Dynamo: Amazon's Highly Available Key-value Store (SOSP 2007) · AWS — Amazon S3 strong consistency announcement (2020 blog + re:Invent talk) · Werner Vogels — Eventually Consistent (CACM 2009) · Calder et al. — Windows Azure Storage: A Highly Available Cloud Storage Service (SOSP 2011) · Plank — A Tutorial on Reed-Solomon Coding for Fault-Tolerance in RAID-like Systems (1997) · MinIO — Erasure-coded object store architecture and docs · Ceph RADOS — paper + docs on the placement group / CRUSH algorithm · Backblaze — How we get 99.99...% durability (engineering blog series)
flat keyspace + bucket + object semantics (no real directories)
immutable objects + versioning + delete markers
replication (3x) vs erasure coding (k+m, Reed-Solomon)
durability math: P(loss) under independent disk failures
read-after-write consistency (the 2020 strong-consistency flip)
multipart upload + per-part ETag + completion manifest
range reads + partial-object cache at the edge
lifecycle policies: hot → infrequent → archive → delete
storage classes: Standard / IA / Glacier (cost vs retrieval latency)
cross-region replication + the strongly-vs-eventually consistent trade
pre-signed URLs + IAM-style request authorization
the per-prefix throughput limit (avoid hot prefixes via sharded keys)