Why a similarity threshold doesn't travel

A team building search over a recipe archive wants the system to return nothing rather than something irrelevant. They look at a few dozen queries, notice that good matches score above a certain similarity and bad ones below it, and set that value as a floor.

It holds for a week. Then a query about a technique returns nothing at all while a query about an ingredient returns five unrelated results, both from the same threshold. Neither query is unusual.

Query: "how long to rest a dough"        top dense score: high band
1. Resting and folding times          ✓
2. Bulk fermentation guide            ~

Query: "sumac"                            top dense score: low band
1. Middle Eastern spice notes          ✓
2. Za'atar blends                      ~
3. Substituting citrus                 ~

Both lists are healthy. The second one’s scores are lower because the query is one rare word, not because its results are worse. A single floor accepts the first and rejects the second.

What a similarity score is comparable to

A cosine similarity is a comparison between two vectors, and its value depends on where both of them sit in the space. That has an unintuitive consequence: the score tells you about the relationship between this query and this document, and almost nothing you can compare across queries.

Three reasons the scale moves under you:

Query length and specificity move the whole band. A long, information-dense query has a vector far from the origin of the space’s typical mass and tends to produce lower similarity to everything, because it is asking for a conjunction of properties no single document fully has. A short, generic query lands in a dense region and scores high against many documents. The band shifts, and the shift has nothing to do with result quality.

Embedding spaces are anisotropic. Trained embeddings do not fill their space uniformly; text tends to occupy a relatively narrow cone. Within that cone, the similarity between two arbitrary unrelated texts is not near zero — it is often substantially positive. That baseline differs by region of the space, so it differs by topic. A threshold calibrated on one topic is measuring against a different baseline on another.

The scale is model-specific and version-specific. Change embedding models and every score changes. Change the model’s version and they change again, usually less, still enough. A threshold is a constant fitted to a model you will eventually replace, and nothing in the pipeline will tell you it has become wrong.

Add the scale incomparability between retrievers and you have the full picture: a raw score is meaningful only inside one query, one model, one retriever.

The failures a fixed threshold produces

Silent over-rejection. The worst one, because it produces no results and no error. Queries about rare topics — the ones where your corpus’s coverage is thin and a good match matters most — score lower and get cut. The system looks like it lacks content it actually has, and nobody debugs a search that returned nothing on the assumption that the answer wasn’t there.

Confident under-rejection. Generic queries score high against many documents, sail over the floor, and the threshold you added specifically to prevent irrelevant answers passes a full page of them.

Drift on reindex. New content shifts nothing about the geometry, but a model upgrade shifts everything, and thresholds are typically committed as a constant in a config file with no test covering them.

What to threshold instead

Four alternatives, roughly in order of how much machinery they require.

Relative gap within the query. Instead of an absolute floor, look at the shape of this query’s own score distribution: how much higher is the top result than the tail? A query whose top score barely exceeds its tenth is a query where nothing stood out, which is a much better proxy for “no good answer” than any absolute value. This is per-query and self-calibrating, and it is the cheapest thing that works.

Agreement between the two signals. If the lexical and dense lists share their top results, something real matched. If they overlap not at all, treat the query as low-confidence. This costs nothing extra in a hybrid pipeline — you have both lists already — and it is a genuinely different kind of evidence from either score.

A calibrated reranker score. A cross-encoder produces a relevance judgement for a query-passage pair directly, and the useful property is that its output is intended to be a relevance estimate rather than a geometric byproduct. It is far more comparable across queries than a similarity is. Not perfectly comparable, and it needs calibrating on your own data, but this is the mechanism most systems that need an abstention decision end up using — one more thing the reranking stage pays for.

A percentile from your own distribution. Log top-1 similarity for every query over a period, then threshold at a percentile of that empirical distribution rather than at a fixed value. This adapts to model changes automatically and encodes an assumption worth being explicit about: that a fixed fraction of your queries are unanswerable.

If you must use an absolute floor

Sometimes you need one anyway — a hard guard against returning something absurd. Three ways to make it less fragile:

Set it far below where you think the boundary is. Use it as a floor against nonsense, not as a relevance decision. The cost of over-rejection is worse than the cost of one weak result, because over-rejection is invisible.

Make it a tested constant, not a config value. A test that asserts a set of known-good queries clear the floor and a set of known-nonsense queries don’t. Then a model upgrade fails a test instead of degrading production.

Log every rejection, with the query and the top scores. A rejected query is the one case where you have no output to inspect later, so it has to be captured at the time. A threshold without rejection logging is a component you cannot diagnose, and it will be blamed for problems it didn’t cause while hiding the ones it did.

Telling whether your threshold is hurting you

One measurement, and it takes an afternoon.

Take a period of query logs, replay them with the threshold disabled, and look at what the threshold would have suppressed. For each suppressed query, was there a relevant passage in the results? The answer will not be “no” uniformly, and the queries where it is “yes” are your over-rejections. Read them and look for a pattern: if they cluster around rare terms, short queries, or one part of your subject matter, the threshold is encoding a bias about your corpus that nobody chose.

Then do the reverse: sample queries that cleared the threshold and had nothing relevant. If that set is large, the floor is not doing the job it was added for, and a per-query gap test or signal agreement will do it better for the same cost.