Search and Full-Text Indexing

When a user types "best tacos near me" into Yelp, they expect results ranked by relevance, not raw database rows. When someone searches for problems on LeetCode, they expect partial matches, tag filters, and difficulty sorting all at once. Relational databases are great at exact matches and range filters, but they fall apart when users expect fuzzy matching, synonym handling, and relevance ranking. That is where full-text search comes in.

The core idea: instead of scanning every row in a table to find matches, you build a specialized data structure — an inverted index — that lets you look up any word and instantly get back every document containing it. The rest of this pattern explains how that index works, how Elasticsearch distributes it across a cluster, how to score results so the best ones come first, and how to keep everything in sync with your primary database.

The Inverted Index — Why Regular Databases Struggle#

Think about how a relational database stores data. Each row is a document, and to find all documents containing the word "caching," the database must scan every single row and check if the text column contains that word. That is a full table scan — O(n) — and it gets slower as your data grows.

An inverted index flips the structure. Instead of mapping documents to their contents, it maps every unique term to the list of document IDs containing that term. Looking up "caching" is now a single hash-table lookup — O(1) — regardless of how many documents you have.

This is exactly the same insight behind database indexes (B-Trees map column values to row IDs), but inverted indexes are designed specifically for text. They handle things B-Trees cannot: partial matches, stemming (matching "running" when searching for "run"), and multi-word queries across multiple fields.

The Analysis Pipeline — From Raw Text to Searchable Tokens#

Raw text cannot go directly into an inverted index. The string "The Redis Caching Guide is GREAT!" needs to be broken down and normalized so that a search for "redis caching" actually matches it. This happens through a configurable analysis pipeline.

The pipeline has four main steps:

1. Tokenization splits text into individual words. "Redis Caching Guide" becomes three separate tokens.
2. Lowercasing normalizes case so "Redis" and "redis" match the same index entry.
3. Stop word removal drops common words like "the," "is," and "at" that appear in almost every document and add noise without improving relevance.
4. Stemming reduces words to their root form. "running," "runs," and "ran" all become "run." This dramatically increases recall — the percentage of relevant documents your search finds.

An optional fifth step, synonym expansion, maps equivalent terms so "NYC" matches "New York City." This is especially useful in e-commerce search where users describe the same product in many different ways.

Each step is configurable per field. A title field might skip stemming to preserve exact brand names, while a body field applies the full pipeline for maximum recall. The trade-off is always the same: more analysis steps mean more matches found (higher recall) but potentially more irrelevant results (lower precision) and a larger index.