Arqly
Sign in
Build a distributed search engine (Elasticsearch / OpenSearch style) (12 scenes)
Scene 12 · Design your search cluster
Capstone: pick e-commerce, logs, or security analytics and configure shards, replicas, refresh, scoring, and ILM — the verifier traces every ✓/✗ back to the scene that earned it.
Previously

The same primitives — shards, replicas, refresh, scoring, tiers — configure radically different deployments. The capstone is choosing them deliberately, with each choice traceable to the scene that justifies it.

Scene 12
Design your search cluster
Diagram
Top: three workload archetype cards (e-commerce search, application logs, security analytics) — exactly one is active. Center: the design palette — primary shards, replicas, refresh interval, dfs_query_then_fetch toggle, ILM tiering depth. Right: the verifier panel — one row per knob, ✓ / ✗ / ! with the scene id that justifies the verdict (scene 4 = refresh, scene 5 = shards, scene 6 = replicas, scene 9 = dfs, scene 10 = aggs, scene 11 = ILM/operational). Bottom: verdict pill — fits the workload, or doesn't.
ACTIVEE-commerce search · 5M produc…Low write rate, high read rate, BM25 critica…ecommerceApplication logs · 10TB/dayWrite firehose, time-series, cheap historica…logsSecurity analytics · 1B event…Aggregations dominate, top-N over source IPs…securityDESIGN PALETTEprimary shards2primary_shardsreplicas1replicasrefresh_interval1srefresh_intervaldfs_query_then_fetchONdfs_query_then_fetchILM tieringnoneilm_tieringVERIFIER✓2 primary shards — fits a 5M catalog↳ scene 5 · small corpus; few shards keep …✓replicas=1 — read fan-out for catalog…↳ scene 6 · replicas scale reads, not writ…✓refresh=1s — interactive product brow…↳ scene 4✓dfs_query_then_fetch ON — rankings st…↳ scene 9 · global IDF; one extra round tr…✓ILM off — product catalog is small an…↳ scene 11✓FITS THE WORKLOADhonest fit for ecommerceevery ✓ / ✗ cites the scene that justifies it
Three workloads, three honest configurations. The canvas snaps to the defaults for each archetype; read the verifier rows — every ✓ cites the scene that justifies it.
Implementation
archetype.ecommerce
small corpus, BM25 critical, dfs ON, ILM off
1# 5M product catalog · interactive browse · ranking is part of SLO
2# primary_shards = 2          (scene 5: small corpus, few shards)
3# replicas       = 1          (scene 6: HA + read fan-out)
4# refresh        = 1s         (scene 4: live catalog UI)
5# search_type    = dfs_query_then_fetch  (scene 9: stable rankings
6#                                          across reindex)
7# terms_agg      = default shard_size    (scene 10: not the bottleneck)
8# ILM            = off        (scene 11: products don't age out)
PreviousOperational sharp edges