What vector search misses

A support engineer searches for ERR_2041. The vector index returns five documents about five different error codes, none of them 2041. The document that documents ERR_2041 exists, is indexed, and is not in the top fifty.

This is not a bug and it’s not a tuning problem. It’s what nearest-neighbour search over embeddings does, and the fix isn’t a better embedding model.

Query: "ERR_2041"

DENSE (vector)
1. Troubleshooting ERR_1180: connection reset     ✗
2. Error code reference: 2000-series              ~
3. Handling ERR_3302 during upload                ✗
4. Common client errors and what they mean        ~
5. ERR_2044: payload too large                    ✗

LEXICAL (BM25)
1. ERR_2041: signature verification failed        ✓
2. Error code reference: 2000-series              ~
3. Changelog 4.2 — new error codes                ~
4. ERR_2044: payload too large                    ✗
5. Migration guide (mentions ERR_2041 once)       ~

The lexical retriever found it because the token err_2041 appears in one document and almost nowhere else — a rare term, which is exactly the situation term-frequency scoring is built to exploit. The dense retriever found things like it, because that’s the only thing it can do.

The mechanism

An embedding maps text to a point in a continuous space where nearby means similar-in-meaning. That mapping is learned from a training corpus, and it’s lossy by design — the whole value of an embedding is that it discards surface form and keeps semantics.

Which means: when surface form is the query, the thing you’re searching by is the thing the embedding threw away.

ERR_2041 and ERR_2044 are semantically near-identical. Both are error codes, both from the same series, both appearing in the same kind of document. A representation that captures meaning will place them close together, and there is no amount of dimensionality that makes a model distinguish two arbitrary integers it has no reason to care about.

Compounding it: rare identifiers usually aren’t in the tokenizer’s vocabulary, so they fragment into subword pieces — ERR, _, 20, 41. The embedding is built from those fragments, which are shared with every other error code. The model isn’t ignoring your identifier; it never saw it as a unit.

Five classes to watch for

Identifiers. SKUs, error codes, model numbers, ticket references, version strings, file paths, function names, ISBNs. Anything where two values differing by one character are entirely different things. This is the largest class and the most common cause of “vector search doesn’t work for us.”

Rare and out-of-domain terms. A proper noun the embedding model has never encountered — an internal project name, a niche technical term, a customer’s company name — lands somewhere generic. Two unfamiliar words can end up close to each other purely because both are unfamiliar.

Exact phrases. When someone quotes a specific sentence to find the document containing it, they want string matching. Dense search returns documents that say something similar, which is precisely not the request.

Negation and small logical words. “Deployments that do not require downtime” and “deployments that require downtime” produce embeddings that are close together, because they share almost all their content words. This one catches people because the query looks like a semantic query — it’s phrased naturally, it isn’t an identifier — and it still fails.

Query: "policies that do not apply to enterprise accounts"

DENSE (vector)
1. Enterprise account policy overview             ✗
2. Policies applying to enterprise accounts       ✗
3. Standard account terms                         ~
4. Enterprise onboarding requirements             ✗
5. Exceptions to the standard policy              ✓

The one relevant result is ranked fifth, and the top two are the opposite of the request. Lexical search does no better here — it also has no notion of negation — which makes this a case where neither signal helps and the fix has to be a query rewrite or a filter.

Numeric and comparative constraints. “Under 500 MB”, “released after 2024”, “more than 50 seats”. Embeddings represent numbers poorly and cannot evaluate comparisons at all. These belong in a metadata filter, not in the similarity query, and no retriever will handle them from text alone.

What this doesn’t mean

It doesn’t mean dense retrieval is weak. On paraphrase — the case where the user’s words and the document’s words don’t overlap at all — it does something lexical search fundamentally cannot, and that case is common enough to justify the whole approach. That’s the other half of this post.

It also doesn’t mean the answer is a better model. Newer embedding models handle rare tokens somewhat better and none of them solve identifier lookup, because the objective they’re trained on is similarity, and your identifier query wants identity. Different problem.

What actually helps

Run a lexical retriever alongside. The direct fix. BM25 handles rare terms well by construction — a term appearing in one document out of a million gets a very high weight. Both retrievers run, the results get merged, and identifier queries are answered by the signal equipped for them.

Detect identifier-shaped queries and route them. If a query is a single token matching a known pattern — a SKU format, an error code prefix — skip semantic retrieval and do an exact lookup. A regex and a dictionary. Crude, effective, and it removes the worst failures without any ranking machinery.

Put identifiers in metadata and filter on them. If your documents have an error_code or sku field, extract it at ingest and match structurally. Retrieval was never the right tool for exact lookup on a known field.

Rewrite negation queries. “Not X” is better served by retrieving X and filtering, or by reformulating into what the user positively wants. Handle it before retrieval, because neither retriever can.

Move numeric constraints into filters. Parse them out of the query and apply them as predicates.

Telling whether it’s your problem

The general answer is to look at your query log, and the specific one is more useful: sample the queries that returned nothing the user clicked, and sort them by how many are a single token.

Query populations vary enormously. A documentation search for developers is full of function names and error codes, and dense-only retrieval will be visibly bad. A search over customer support narratives is mostly paraphrase, and lexical-only retrieval will be visibly bad. The mix determines which failures you actually have, so measure the mix rather than assuming.

A quick diagnostic that costs almost nothing: take fifty real queries, run them through both retrievers, and mark which one found the right document. If a distinct category shows up in the lexical-only column, you’ve just identified the class of query your system currently can’t answer — and probably which signal to blame when it doesn’t.