Build a graph database (Neo4j / Dgraph-style)
16 scenes · ~112 min · build the primitive

Build your own graph database (Neo4j / Dgraph-style)

When the workload is 'friends of friends', a relational join melts. Build a store where edges are first-class — index-free adjacency, traversals that follow pointers instead of joining tables, and a query language (Cypher / GraphQL+) that thinks in patterns. Feel why graph storage shines for traversal-heavy work and stumbles on full-graph aggregates.

Scenes
16 interactive scenes
Time
about 112 minutes
Topic
Storage Engines & Databases

What you are building, and why

You are building the simplest thing that could possibly answer one question: "how is this connected to that?" Friends of friends, the parts that make up an assembly, the accounts a fraud ring routes money through, the page that links to the page that links to yours. The shape is always the same — entities joined by relationships — and the query is almost always "walk the relationships and tell me what you reach."

The obvious plan is a relational one: a USER_FRIEND(user, friend) join table. It works beautifully for one hop. But "friends of friends of friends" is the same table self-joined, once per hop, and the intermediate result explodes combinatorially. With a million users averaging 50 friends each, a depth-4 query materializes on the order of 6.25 million candidate rows (depth-5: ~312 million) that the engine must build, sort, and dedupe. In the classic Neo4j in Action benchmark, that depth-4 query ran in ~1.3 s on a graph store while MySQL took ~1,543 s — and could not finish depth 5 at all. The join didn't get slow; it melted.

So you change the storage, not the query. You make edges first-class: every node stores a direct pointer to its own relationship records, so following an edge is a pointer dereference — roughly O(1), and independent of how big the graph is — instead of a B-tree index seek that pays O(log N) on every hop and gets slower as the data grows. That one trick is index-free adjacency, and it is the entire reason this category of database exists. The tagline for the whole course: follow pointers, don't join tables.

The honest other half — which you will build toward and feel directly — is that the same design has no locality for whole-graph work. A k-hop traversal lights up a few nodes; PageRank lights up every node, every pass. Global aggregates have nothing for index-free adjacency to exploit, and you fall back to the Pregel / "think like a vertex" world of supersteps and barriers. Resist the urge to "describe Neo4j." Build the store yourself — the records, the pointer chains, the pattern compiler, the locks, the partition cut — and you will know exactly where a graph database shines and exactly where it stumbles.

What you will be able to explain afterwards

  • property graph model: nodes, edges, labels, properties
  • index-free adjacency — each node stores its edge list
  • edge stores: doubly-linked lists vs adjacency arrays vs compressed CSR
  • traversal-as-pointer-chase (no join planner needed for k-hop)
  • pattern matching: Cypher MATCH, SPARQL triples, GraphQL+
  • distributed graphs: vertex-cut vs edge-cut partitioning
  • the supernode problem (one celebrity, 10M edges)
  • global indexes: full-text, range, vector — bolted on, not native
  • ACID transactions over a graph (Neo4j) vs eventual (Dgraph)
  • graph algorithms layer: BFS, shortest path, PageRank
  1. 01
  2. 02
  3. 03
  4. 04
  5. 05
  6. 06
  7. 07
  8. 08
  9. 09
  10. 09a
  11. 10
  12. 11
  13. 11a
  14. 12
  15. 13
  16. 14

Why graphs

Friends-of-friends melts a join; edges go first-class.

  1. 01
    Friends of friends melts a join — the pointer walk that replaces the self-join
    The canonical graph workload — 'friends of friends of friends' — is one self-join per hop, and the intermediate row count explodes combinatorially until a relational engine can't finish; the same query as a pointer walk visits only the people you actually reach.
    ~7 min
  2. 02
    Nodes, relationships, properties — first-class relationships versus RDF reification and junction tables
    The property-graph model is four things — nodes, typed relationships, labels, and properties on both — and putting key/values directly ON the edge is the move that relational join-tables and RDF triples can't make cleanly.
    ~7 min
  3. 03
    Index-free adjacency: follow pointers
    Each node stores a direct pointer to its relationships, so following an edge is a pointer dereference — O(1) per hop, independent of total graph size — not a B-tree seek that gets slower as the database grows.
    ~7 min

The store

Store records, index-free adjacency, the CSR trade.

  1. 04
    Store records and the relationship chain
    Fixed-size store records (node ~15 B, relationship ~34 B) mean record-id × size = byte offset with no index, and each relationship sits in a doubly-linked list for BOTH its endpoints — so a node's edges are a chain you can walk, insert into, and delete from in place.
    ~7 min
  2. 05
    You still need an index to start — index-free adjacency covers EXPAND, not SEEK
    Index-free adjacency only covers EXPAND (traversing from a node you hold); to SEEK the anchor — 'start from Alice' — you still need a regular index, and without one finding the start node is a full label scan over every :User.
    ~7 min
  3. 06
    Linked list vs CSR adjacency
    Doubly-linked relationship lists are cache-hostile but mutate in place; Compressed Sparse Row packs a node's neighbors into one contiguous slice that streams through cache but rebuilds the whole array to add a single edge — the OLTP-graph vs OLAP-graph storage fork.
    ~7 min

The language

Cypher is a shape; the planner picks an anchor.

  1. 07
    Cypher: describe a shape
    A Cypher MATCH (a:User {name:'Alice'})-[:FRIEND]->(b)-[:FRIEND]->(c) is a declarative description of a pattern, and for expansion it maps directly onto pointer-chasing — no join planner needed.
    ~7 min
  2. 08
    Anchor selection and cardinality
    The planner's real job is to anchor on the most selective node and estimate cardinality — start from the few, not the many — so flipping a WHERE filter can flip which end is cheapest and reverse the whole traversal direction.
    ~7 min

It breaks

The supernode shatters the O(1)-per-hop promise.

  1. 09
    The supernode breaks the promise
    A supernode is a vertex with a huge degree, and its relationship chain is so long that any traversal THROUGH it must scan millions of edges — so the O(1)-per-hop promise dies on that one node, and the fix is to expand from the low-degree side.
    ~7 min
  2. 09a
    Don't block The Rock — relationship-chain locks and dense-node grouping for supernodes
    Concurrent edge-adds to a supernode historically serialized on a whole-node lock; relationship-chain locks plus dense-node grouping let writers touch different parts of the chain at once, turning a serialized hotspot into parallel throughput.
    ~7 min

Scale & ACID

ACID on one box; the partition cut turns hops to RPCs.

  1. 10
    ACID on a single primary
    A single-primary graph store gives full ACID — a write-ahead logical log for durability, write locks held to commit, and deadlock detection via a wait-for graph — at the cost of write throughput bounded by one machine.
    ~7 min
  2. 11
    The cut turns hops into RPCs — graph sharding, locality and network round trips
    Split a graph across two machines and any cut severs edges, so a traversal that crosses the cut turns each pointer dereference into a network round trip — the boundary is exactly where O(1) becomes O(network).
    ~7 min
  3. 11a
    Edge-cut, vertex-cut, predicate sharding
    Edge-cut assigns each vertex to a machine and chokes on power-law hubs; vertex-cut splits the hub across machines to balance edges; predicate-sharding keeps all edges of one TYPE together so a common expand stays on one machine.
    ~7 min
  4. 12
    Search indexes bolted on the side — secondary indexes for full-text, range and vector seeks
    Index-free adjacency only serves EXPAND, so finding start nodes by value — full-text, range, geo, vector — is served by classic secondary indexes (Lucene, B-tree) maintained as separate structures riding shotgun, with the usual write-amplification and staleness costs.
    ~7 min

The payoff

No locality for whole-graph work — then design it.

  1. 13
    Think like a vertex: Pregel / BSP
    Whole-graph algorithms have no locality to exploit, so the model flips to Pregel/BSP — every vertex runs a small function, sends messages to neighbors, then all vertices wait at a synchronization barrier before the next superstep — and a supernode at the barrier stalls everyone.
    ~7 min
  2. 14
    Design your graph database
    Every graph-DB deployment is a deliberate set of choices — storage layout, index strategy, supernode handling, single-node ACID vs distributed, and traversal-heavy vs aggregate-heavy workload — and the right configuration for a fraud-ring traversal is wrong for a PageRank pipeline even though the primitives are identical.
    ~7 min

More in Storage Engines & Databases

Open the box every design diagram labels "DB": pages, logs, LSM trees, wide-column, documents, graphs and columnar scans, built from scratch.

Prefer to design it yourself?

The same subject as a staged workspace: draw the architecture, and a simulator traces requests through the boxes you drew.

Open the Build a graph database (Neo4j / Dgraph-style) workspace