Two Signals

The analyser decides what you can match

A codebase search cannot find parseUserRecord. The function is in the indexed source, the query is the exact identifier, and lexical retrieval returns nothing. Meanwhile a search for user returns it along with four thousand other files.

The scoring function never got a chance. By the time BM25 saw the query, parseUserRecord had become three tokens — parse, user, record — and the document’s rarest, most discriminating string had stopped existing.

Query: "parseUserRecord"

ANALYSER SPLITS camelCase                 ANALYSER KEEPS camelCase
1. user_service.py                    ~   1. record_parser.py                 ✓
2. record_store.py                    ✗   2. user_import.py (calls it)        ~
3. parse_utils.py                     ~   3. CHANGELOG (mentions it)          ~
4. user_import.py                     ~   4. user_service.py                  ✗
5. record_parser.py                   ✓   5. parse_utils.py                   ✗

The pipeline in front of the scoring function

Lexical retrieval is two things, and people talk about it as one. There is a scoring function, and in front of it there is a text analysis pipeline that decides what the units of matching are. Both the document and the query pass through it — and they must pass through compatible versions of it, or nothing matches at all.

raw text → character filters → tokeniser → token filters → terms
                                              (lowercase, stem,
                                               stop words, synonyms)

Every claim about “what keyword search can find” is really a claim about this pipeline. BM25 will faithfully score whatever tokens it is handed. If the tokens are wrong, the ranking is wrong in ways no parameter fixes, and the failure looks like the retriever ignoring text that is visibly present.

The tokeniser decides the hard cases

Tokenisation looks trivial for prose and is decisive everywhere else.

Identifiers with internal punctuation. ERR_2041, TX-9910, SKU/44-7. A tokeniser splitting on non-alphanumeric characters turns each into fragments shared with every other identifier in the corpus, and the rare-term advantage that made identifier queries lexical retrieval’s specialty disappears. This is the single most consequential analyser decision in most technical corpora, and it is usually made by accident, by whichever default was in place.

camelCase and snake_case. Splitting them helps someone searching parse user; it hurts someone searching the exact symbol. There is no setting that serves both, which is why the usual answer is to index both forms — emit the whole token and its parts — and accept a larger index.

Hyphenation and compounds. “e-mail” against “email”, “twenty-four” against “twentyfour”. Whether these match depends entirely on configuration, and users type both.

Languages without whitespace. Chinese, Japanese, Thai. A whitespace tokeniser produces one enormous token per sentence, and the index is silently useless. This fails so completely that it is usually caught, but it fails only for those documents, so in a mixed-language corpus it can pass unnoticed while the queries in those languages all fail.

Numbers and units. “500MB” as one token, or “500” and “MB”, or “500”, “M”, “B”. Each choice makes a different set of queries answerable.

Stemming: what it fixes and what it breaks

Stemming reduces inflected forms to a common root so “converting” matches “conversion”. It buys real recall on ordinary prose, and it has three costs worth stating.

It conflates distinct terms. Aggressive stemmers map words with different meanings to the same root. The classic pattern is a technical term stemmed into a common word, after which a precise query returns everything.

It cannot be reversed at query time. Once the index stores stems, the exact form is gone. A user who wants the exact word cannot ask for it, so a phrase or exact-match feature needs an unstemmed field alongside — which is the standard arrangement and is worth building deliberately rather than discovering later.

Its effect is language-specific and quality varies. A stemmer tuned for English does something arbitrary to other languages, and lemmatisation — the more accurate, dictionary-based alternative — costs more and needs per-language resources.

Note what stemming is not: it does not handle synonyms, and it does not repair misspellings. Teams reach for a more aggressive stemmer to solve vocabulary mismatch, and it is the wrong tool — that problem belongs to expansion or to the dense signal.

Stop words, and the query made entirely of them

Removing very common words was originally an index-size optimisation, and rarity weighting already handles most of what it was for — a term appearing everywhere gets a low weight anyway.

Where removal still bites is on queries whose meaning lives in the removed words:

  • “to be or not to be” — every token removed, empty query.
  • “vitamin A” — a single-letter token dropped as noise.
  • “the who”, “it” as a title, “how to” phrases where “how” carries the intent.
  • Negation: “not covered” becomes “covered”, which is an inversion, not a simplification.

A query that analyses to zero terms returns nothing, and the log shows a query with no results and no error. Worth an explicit check: if analysis empties the query, fall back to the unfiltered form rather than returning an empty list.

The rule that catches most of it

The query and the document must be analysed compatibly, and the failure mode is silence.

If documents are stemmed and queries are not, terms mismatch and results are thin without an error. If documents were indexed under one analyser and the configuration later changed, everything indexed before the change is matched under different rules than everything after — and no reindex means no correction. An analyser change is a reindex, and treating it as a config tweak produces a corpus with two incompatible halves.

How to find out what yours does

Most engines expose an endpoint or command that shows the token stream for a given input. It is the single most useful diagnostic in lexical retrieval and it is chronically unused.

Build a fixture list from your own corpus — your identifier formats, your unit strings, your domain terms, a hyphenated compound, a camelCase symbol, a query that is all stop words — and assert the token stream for each in your test suite. Then an analyser change fails a test instead of silently changing which queries are answerable.

When diagnosing a lexical miss, check the token stream before anything else. It costs one command, and it distinguishes “the scoring ranked it badly” from “the term does not exist in the index”, which are different problems with nothing in common.

How much any of this matters depends on your corpus. Prose in one language with few identifiers is well served by defaults. Source code, part catalogues, legal citations, chemical names, or anything multilingual is not, and in those corpora the analyser is where a surprising share of retrieval quality is decided — before a single score is computed.

lexical