Build your own graph database (Neo4j / Dgraph-style)
When the workload is 'friends of friends', a relational join melts. Build a store where edges are first-class — index-free adjacency, traversals that follow pointers instead of joining tables, and a query language (Cypher / GraphQL+) that thinks in patterns. Feel why graph storage shines for traversal-heavy work and stumbles on full-graph aggregates.
- Scenes
- 16 interactive scenes
- Time
- about 112 minutes
- Topic
- Storage Engines & Databases
What you are building, and why
You are building the simplest thing that could possibly answer one question: "how is this connected to that?" Friends of friends, the parts that make up an assembly, the accounts a fraud ring routes money through, the page that links to the page that links to yours. The shape is always the same — entities joined by relationships — and the query is almost always "walk the relationships and tell me what you reach."
The obvious plan is a relational one: a USER_FRIEND(user, friend) join table. It works beautifully for one hop. But "friends of friends of friends" is the same table self-joined, once per hop, and the intermediate result explodes combinatorially. With a million users averaging 50 friends each, a depth-4 query materializes on the order of 6.25 million candidate rows (depth-5: ~312 million) that the engine must build, sort, and dedupe. In the classic Neo4j in Action benchmark, that depth-4 query ran in ~1.3 s on a graph store while MySQL took ~1,543 s — and could not finish depth 5 at all. The join didn't get slow; it melted.
So you change the storage, not the query. You make edges first-class: every node stores a direct pointer to its own relationship records, so following an edge is a pointer dereference — roughly O(1), and independent of how big the graph is — instead of a B-tree index seek that pays O(log N) on every hop and gets slower as the data grows. That one trick is index-free adjacency, and it is the entire reason this category of database exists. The tagline for the whole course: follow pointers, don't join tables.
The honest other half — which you will build toward and feel directly — is that the same design has no locality for whole-graph work. A k-hop traversal lights up a few nodes; PageRank lights up every node, every pass. Global aggregates have nothing for index-free adjacency to exploit, and you fall back to the Pregel / "think like a vertex" world of supersteps and barriers. Resist the urge to "describe Neo4j." Build the store yourself — the records, the pointer chains, the pattern compiler, the locks, the partition cut — and you will know exactly where a graph database shines and exactly where it stumbles.
What you will be able to explain afterwards
- property graph model: nodes, edges, labels, properties
- index-free adjacency — each node stores its edge list
- edge stores: doubly-linked lists vs adjacency arrays vs compressed CSR
- traversal-as-pointer-chase (no join planner needed for k-hop)
- pattern matching: Cypher MATCH, SPARQL triples, GraphQL+
- distributed graphs: vertex-cut vs edge-cut partitioning
- the supernode problem (one celebrity, 10M edges)
- global indexes: full-text, range, vector — bolted on, not native
- ACID transactions over a graph (Neo4j) vs eventual (Dgraph)
- graph algorithms layer: BFS, shortest path, PageRank
Why graphs
Friends-of-friends melts a join; edges go first-class.
- 01Friends of friends melts a join — the pointer walk that replaces the self-joinThe canonical graph workload — 'friends of friends of friends' — is one self-join per hop, and the intermediate row count explodes combinatorially until a relational engine can't finish; the same query as a pointer walk visits only the people you actually reach.~7 min
- 02Nodes, relationships, properties — first-class relationships versus RDF reification and junction tablesThe property-graph model is four things — nodes, typed relationships, labels, and properties on both — and putting key/values directly ON the edge is the move that relational join-tables and RDF triples can't make cleanly.~7 min
- 03Index-free adjacency: follow pointersEach node stores a direct pointer to its relationships, so following an edge is a pointer dereference — O(1) per hop, independent of total graph size — not a B-tree seek that gets slower as the database grows.~7 min
The store
Store records, index-free adjacency, the CSR trade.
- 04Store records and the relationship chainFixed-size store records (node ~15 B, relationship ~34 B) mean record-id × size = byte offset with no index, and each relationship sits in a doubly-linked list for BOTH its endpoints — so a node's edges are a chain you can walk, insert into, and delete from in place.~7 min
- 05You still need an index to start — index-free adjacency covers EXPAND, not SEEKIndex-free adjacency only covers EXPAND (traversing from a node you hold); to SEEK the anchor — 'start from Alice' — you still need a regular index, and without one finding the start node is a full label scan over every :User.~7 min
- 06Linked list vs CSR adjacencyDoubly-linked relationship lists are cache-hostile but mutate in place; Compressed Sparse Row packs a node's neighbors into one contiguous slice that streams through cache but rebuilds the whole array to add a single edge — the OLTP-graph vs OLAP-graph storage fork.~7 min
The language
Cypher is a shape; the planner picks an anchor.
- 07Cypher: describe a shapeA Cypher MATCH (a:User {name:'Alice'})-[:FRIEND]->(b)-[:FRIEND]->(c) is a declarative description of a pattern, and for expansion it maps directly onto pointer-chasing — no join planner needed.~7 min
- 08Anchor selection and cardinalityThe planner's real job is to anchor on the most selective node and estimate cardinality — start from the few, not the many — so flipping a WHERE filter can flip which end is cheapest and reverse the whole traversal direction.~7 min
It breaks
The supernode shatters the O(1)-per-hop promise.
- 09The supernode breaks the promiseA supernode is a vertex with a huge degree, and its relationship chain is so long that any traversal THROUGH it must scan millions of edges — so the O(1)-per-hop promise dies on that one node, and the fix is to expand from the low-degree side.~7 min
- 09aDon't block The Rock — relationship-chain locks and dense-node grouping for supernodesConcurrent edge-adds to a supernode historically serialized on a whole-node lock; relationship-chain locks plus dense-node grouping let writers touch different parts of the chain at once, turning a serialized hotspot into parallel throughput.~7 min
Scale & ACID
ACID on one box; the partition cut turns hops to RPCs.
- 10ACID on a single primaryA single-primary graph store gives full ACID — a write-ahead logical log for durability, write locks held to commit, and deadlock detection via a wait-for graph — at the cost of write throughput bounded by one machine.~7 min
- 11The cut turns hops into RPCs — graph sharding, locality and network round tripsSplit a graph across two machines and any cut severs edges, so a traversal that crosses the cut turns each pointer dereference into a network round trip — the boundary is exactly where O(1) becomes O(network).~7 min
- 11aEdge-cut, vertex-cut, predicate shardingEdge-cut assigns each vertex to a machine and chokes on power-law hubs; vertex-cut splits the hub across machines to balance edges; predicate-sharding keeps all edges of one TYPE together so a common expand stays on one machine.~7 min
- 12Search indexes bolted on the side — secondary indexes for full-text, range and vector seeksIndex-free adjacency only serves EXPAND, so finding start nodes by value — full-text, range, geo, vector — is served by classic secondary indexes (Lucene, B-tree) maintained as separate structures riding shotgun, with the usual write-amplification and staleness costs.~7 min
The payoff
No locality for whole-graph work — then design it.
- 13Think like a vertex: Pregel / BSPWhole-graph algorithms have no locality to exploit, so the model flips to Pregel/BSP — every vertex runs a small function, sends messages to neighbors, then all vertices wait at a synchronization barrier before the next superstep — and a supernode at the barrier stalls everyone.~7 min
- 14Design your graph databaseEvery graph-DB deployment is a deliberate set of choices — storage layout, index strategy, supernode handling, single-node ACID vs distributed, and traversal-heavy vs aggregate-heavy workload — and the right configuration for a fraud-ring traversal is wrong for a PageRank pipeline even though the primitives are identical.~7 min
More in Storage Engines & Databases
Open the box every design diagram labels "DB": pages, logs, LSM trees, wide-column, documents, graphs and columnar scans, built from scratch.
- Build Build a Bitcask-style KV storeThe simplest possible KV store that still works: an append-only log on disk + an in-memory hash index. Build it from first principles and feel which trade-offs every later store inherits.
- Build Build an LSM-tree storage engine (LevelDB / RocksDB style)The simplest possible storage engine that gives you BOTH ordered reads AND more keys than fit in RAM, by accepting a deal: write to RAM at memory speed, log to disk for safety, then merge sorted files in the background forever.
- Build Build a B-tree storage engine (SQLite-style)What actually happens when you run INSERT INTO users(...). One file of fixed-size pages, organized as B-trees, with a write-ahead log that turns commits into appends. Build it from a SQL writer's perspective and feel why every knob exists.
- Build Build a wide-column store (Cassandra / DynamoDB family)One server is not enough — disk fills, throughput maxes, the box dies. Build a multi-node store from first principles: hash sharding, the consistent-hash ring, vnodes, replication, eventual consistency, tunable W+R quorum, hinted handoff, read repair. Every modern Dynamo-style store is a point in this design space.
- Build Build a columnar OLAP store (ClickHouse / Druid style)OLTP picks one row by key; OLAP scans a billion rows of one column and asks for a percentile. Build the analytical engine that makes that fast: columnar layout, dictionary/RLE/delta compression, vectorized execution, late materialization, MPP shuffle. Internalize why Postgres is 1000× slower than ClickHouse on the same query and why the inverse is also true.
Prefer to design it yourself?
The same subject as a staged workspace: draw the architecture, and a simulator traces requests through the boxes you drew.
Open the Build a graph database (Neo4j / Dgraph-style) workspace