What keyword search misses

A user searches for “why is the site slow.” The document that answers it is titled Diagnosing high p99 latency. It never uses the word “slow.”

BM25 returns nothing useful, and it isn’t misconfigured. It scored the query terms against the documents and found that the term “slow” appears in a blog post about slow-cooker recipes in the marketing corpus, and nowhere else.

Query: "why is the site slow"

LEXICAL (BM25)
1. Slow-cooker content marketing case study       ✗
2. Site map and navigation guide                  ✗
3. Why we chose our hosting provider              ✗
4. Site reliability overview                      ~
5. (nothing else scores)                          —

DENSE (vector)
1. Diagnosing high p99 latency                    ✓
2. Performance troubleshooting checklist          ✓
3. Understanding response time percentiles        ~
4. CDN cache miss investigation                   ~
5. Site reliability overview                      ~

This is the vocabulary mismatch problem, and it’s the reason dense retrieval exists.

The mechanism

Lexical scoring functions — BM25 and its relatives — score a document by the query terms it contains, weighted by how rare each term is in the corpus and how often it appears in the document, with a correction for document length.

Every part of that operates on terms. Not concepts, not intent — the tokens that survived the analysis pipeline. A term that isn’t in the document contributes nothing, no matter how obviously related it is. “Slow” and “latency” are unrelated strings. There is no path from one to the other inside the scoring function.

The user and the document author are different people with different vocabularies, writing at different times for different reasons. Expecting their word choices to overlap is the assumption that breaks.

Where the analyser helps, and where it stops

Lexical search isn’t pure string matching. There’s a text analysis pipeline in front of the scorer, and it closes some of the gap:

Tokenisation splits text into terms and normalises punctuation and case, so Latency, latency, and latency. all match.

Stemming or lemmatisation reduces inflected forms to a common root, so “running”, “runs” and “ran” can match “run”. This handles a genuine and common class of mismatch.

Stop-word removal drops high-frequency words that carry little signal.

Synonym expansion, if configured, maps terms to a curated list — slow → latency, latencies, lag.

That last one is the interesting case, because it looks like a solution and isn’t. Synonym lists are hand-maintained: someone has to know in advance that users say “slow” when documents say “latency”. They don’t generalise, they need updating as vocabulary drifts, and they get large and contradictory. They work well for a narrow, stable domain with known jargon — medical coding, legal terminology, a product catalogue — and poorly for anything open-ended.

What stemming and synonyms explicitly cannot do:

Paraphrase. “How do I stop people from signing up twice” versus “preventing duplicate account registration.” No shared content terms, no stem relationship, and no synonym list would have predicted this pairing.

Conceptual queries. “What should I worry about before launching” against a document titled Pre-release checklist. The document is exactly right and shares no vocabulary with the question.

Cross-lingual matching. A query in one language against documents in another. Some embedding models handle this natively; a term index cannot without translation.

Intent behind a description. Users describe symptoms; documentation describes causes. “The button does nothing” versus “event handler not bound.” This mismatch is systematic in support corpora — the person searching doesn’t know the vocabulary yet, which is why they’re searching.

The second failure: everything matches

The mirror image of finding nothing is finding too much, and it’s easier to miss because results come back.

Query: "how do I set up authentication for the API"

LEXICAL (BM25)
1. API reference: overview                        ✗
2. Setting up your API keys                       ~
3. API rate limits and quotas                     ✗
4. Authentication concepts                        ~
5. Setting up the CLI                             ✗

Every document in an API documentation corpus contains “API” and “set up”. Those terms are common, so BM25 down-weights them correctly, but with common terms doing most of the work the scores compress and ranking becomes close to arbitrary. The document that actually walks through API authentication setup is somewhere in the results, ranked on essentially a coin flip.

The general shape: lexical scoring is strong when query terms are rare and weak when they’re common. Rare terms give it something to discriminate on. Generic natural-language questions give it almost nothing — which is unfortunate, because generic natural-language questions are what people type into a chat interface.

Two more, briefly

Word order and proximity are mostly ignored. Standard BM25 is a bag of words: “dog bites man” and “man bites dog” score identically. Phrase and proximity queries exist as features in most engines, and they require the user — or your query builder — to ask for them explicitly.

Morphologically rich and unsegmented languages are harder. Stemming assumes a language where inflection is a suffix you can strip. Languages with heavy compounding, or no whitespace between words, need language-specific analysers, and a default analyser on such a corpus performs badly without saying so.

What actually helps

Run a dense retriever alongside. The direct fix, and the reason hybrid retrieval is the usual answer. Vector search handles paraphrase by construction, because it compares meaning rather than terms.

Query expansion. Generate alternative phrasings of the query — with a model or a thesaurus — and search for all of them. This helps lexical retrieval specifically, since it manufactures the term overlap that was missing. It costs a generation step, adds latency, and can drift the query away from what was asked.

Index generated paraphrases. At ingest, generate a few likely questions each document answers and index those alongside the text. It converts a paraphrase problem into a term-matching problem. Costs a model call per document, and it re-runs whenever you change the generation prompt.

Learned sparse retrieval. Models that produce a sparse term-weighted representation including terms not present in the original text — a document about latency can be given weight on “slow”. You keep the efficiency and interpretability of an inverted index while getting some semantic expansion. A genuine middle path, and it needs a model in the indexing path.

Domain synonym lists, where the domain is narrow. Unfashionable, still effective in exactly the conditions described above.

Telling whether it’s your problem

Look at query length. Short, term-heavy queries suit lexical retrieval; long natural-language questions do not. A search box where people type two or three words has different failure modes from a chat interface where they type sentences, and the shift from boxes to chat is why vocabulary mismatch has become the dominant failure.

The cheap diagnostic is the same as for the reverse case: take fifty real queries, run both retrievers, and mark which one found the right document. A dense-only column full of conversational questions tells you your users’ vocabulary and your corpus’s vocabulary have diverged.

And notice what the two posts together imply. The query classes here and the ones in its counterpart barely intersect. That non-overlap is the whole argument for running both — which raises the next question, how to merge two ranked lists that don’t agree.