rgoussu.dev
rgoussu@goussu: ~/library/system-design
❯ ~/library/system-design cat search-systems.md

Search systems

# How full-text search works — inverted indexes, analysis, relevance scoring, and operating Elasticsearch-class systems next to your database.

Conceptsaved 2026-08-08 #search#elasticsearch#inverted-index#relevance#system-design

Overview

Search is what databases are bad at: ranked, fuzzy, language-aware matching over text. The engine underneath (Lucene, powering Elasticsearch/OpenSearch/Solr) is built on one data structure — the inverted index — plus an analysis pipeline that decides what a "term" is, and a scoring model that decides what "relevant" means. Operating one is a second copy of your data with its own consistency, mapping, and capacity story; knowing the internals is what keeps that copy useful.

Key points

  • Inverted index: term → posting list of documents (plus positions for phrases); queries intersect/union posting lists instead of scanning documents. Immutable segments, merged in the background — a very LSM-shaped design, and the reason updates are really delete + reinsert and search is near-real-time (refresh interval), not transactional.
  • Analysis decides everything: tokenization → lowercasing → stemming/lemmatization, stop words, synonyms, n-grams (for autocomplete), language-specific analyzers. Most "search returns garbage" bugs are analyzer mismatches between index time and query time, not scoring problems.
  • Relevance: TF-IDF's successor BM25 as the standard lexical score (term frequency, saturating; rarity; length normalization), combined with field boosts and function scores (recency, popularity). Measure with click-through and judged sets — relevance tuning without measurement is superstition.
  • Beyond lexical: vector/semantic search (embeddings + ANN indexes like HNSW) catches meaning where BM25 catches words; production systems increasingly run hybrid (BM25 + vector, fused) — also the retrieval layer under RAG (agents & RAG).
  • Operating it: the search cluster is a derived store — feed it via CDC or the outbox pattern, plan for full reindex (mappings are mostly immutable; use index aliases for zero-downtime swaps), size shards deliberately (over-sharding is the classic self-inflicted wound), and don't use it as a primary database.
  • Query-side vocabulary: filters (cacheable, unscored) vs. queries (scored), aggregations/facets, pagination pitfalls (deep offset is O(n) — use search-after), highlighting.
  • To explore: Postgres full-text search as the "good enough" tier (tsvector/GIN), typo-tolerant engines (Typesense, Meilisearch), learning-to-rank, RRF for hybrid fusion.

Practice

  • Analyzer archaeology (source) — feed the _analyze API a handful of real queries and watch tokenization, stemming, and n-grams reshape them; most "search returns garbage" bugs become visible right here.
  • Full-text search engine from scratch (source) — inverted index over a real corpus (Wikipedia abstracts work well): tokenize, build posting lists, intersect them for queries, then add BM25 ranking and n-gram autocomplete.
  • Hybrid retrieval (source) — bolt vector search onto the lexical engine: embed the corpus, build an HNSW index, fuse BM25 + ANN results, and measure whether hybrid actually wins on your queries.

Related