Two Signals

When every document looks the same to the vector index

A managed-services firm indexes four thousand client onboarding runbooks. They are built from one template: same headings, same sections, same phrasing, with the client name and a handful of parameters differing. A query about one client’s firewall exception returns five runbooks, none of them that client’s, all nearly identical in score.

This is not a tuning failure and it is not a model failure. The corpus provides almost no contrast, and a retriever cannot manufacture a distinction the text does not contain.

Query: "firewall exception for Meridian's VPN concentrator"

DENSE (vector)                            LEXICAL (BM25)
1. Onboarding runbook — Caldwell      ✗   1. Onboarding runbook — Meridian     ✓
2. Onboarding runbook — Ashgrove      ✗   2. VPN concentrator standard build   ~
3. Onboarding runbook — Meridian      ✓   3. Firewall exception request form   ~
4. VPN concentrator standard build    ~   4. Onboarding runbook — Caldwell     ✗
5. Onboarding runbook — Northvale     ✗   5. Onboarding runbook — Ashgrove     ✗

The lexical retriever found the right document because “Meridian” is a rare token appearing in one runbook. The dense retriever placed all four thousand runbooks in essentially the same location, and which three came back is close to arbitrary.

What “low contrast” means at the retrieval layer

Dense retrieval orders documents by distance from the query. That ordering is only informative if the distances differ meaningfully. When a corpus consists of documents that are ninety-five per cent identical text, their embeddings cluster tightly, and the differences between their distances to any query are small relative to the noise in the representation.

The retriever is still returning its nearest neighbours. There is simply no meaningful difference between the tenth-nearest and the four-hundredth. The consequences follow directly:

  • Ranking within the cluster is unstable. Small changes — a reindex, a model version, a slightly reworded query — reshuffle the order, because the order was never resting on much.
  • Score-based cut-offs are useless here. Every candidate scores in a narrow band, so a similarity threshold either admits all of them or none.
  • Depth stops helping. Normally retrieving more candidates raises the chance the right one is present. On a homogeneous corpus you can retrieve a hundred near-identical documents and still not have the one you need, because inclusion is close to random with respect to the distinction you care about.

That last point is the one that costs teams time, because deepening retrieval is the standard first response and it is the one intervention that genuinely does not apply.

Corpora that have this shape

Recognising the pattern is most of the fix, because the corpus types are specific:

  • Template-generated documents. Runbooks, per-client procedures, per-region policy copies, generated reports.
  • Form-based records. Insurance claims, incident tickets, inspection reports, application forms — identical structure, differing field values.
  • Highly regular technical reference. Per-endpoint API pages, per-part datasheets, per-drug monographs. Same sections, same prose, different values.
  • Legal and contractual boilerplate. Agreements from one template, where the negotiated deltas are what anyone ever searches for.
  • Single-topic corpora. A corpus entirely about one narrow subject. Everything is topically similar because the topic is the corpus, so topical similarity carries no information.

The unifying property: the thing that distinguishes documents is not the thing dense retrieval represents. Embeddings capture what text is about, these documents are all about the same thing, and they differ in identifiers, names, dates and values — the categories embeddings represent least well.

What still discriminates

Four things, in order of how much they help.

Rare tokens, via the lexical signal. A client name, a part number, a case reference. This is exactly the situation term rarity weighting was built for, and on a low-contrast corpus the lexical retriever is not a fallback — it is the primary signal, and the dense list is the one contributing little.

Which has a direct consequence for fusion: a merge that weights both signals equally is averaging a useful list with a near-random one. This is one of the few situations where a strong weighting toward one signal is a finding rather than a guess — and the honest version of the conclusion is that you should verify the dense list is contributing anything at all before continuing to pay for it.

Structured metadata and predicates. If documents differ by client, region, date, version or status, those differences belong in fields, and the query’s constraint belongs in a predicate. One equality test on a client field beats every ranking strategy in this post. On a template corpus, most queries are secretly filter queries with a search attached.

Required terms. If the distinguishing token must be present, say so — a hard requirement removes the four thousand runbooks that lack it, and an empty result is far better than a confident wrong client’s document.

A cross-encoder, partially. A reranker reads query and passage together and can notice that the passage names a different client, so it does better than similarity within the cluster. Two limits: it can only reorder what retrieval supplied, and on a low-contrast corpus that supply is the problem. Reranking a hundred near-identical candidates is expensive and still fails if the right one wasn’t among them.

Diagnosing it directly

You do not need labels for this, and the measurement is worth running on any corpus before designing retrieval for it.

Sample pairs and look at the distribution of similarities between them. Take a few thousand random document pairs, compute pairwise similarity, and look at the spread. A diverse corpus produces a wide distribution. A low-contrast corpus produces a narrow one concentrated high. You are looking at whether your corpus offers any dynamic range at all.

Compare that to query-document similarities. If the similarity between a query and its correct document sits inside the range of similarities between arbitrary document pairs, then the signal you are ranking on is smaller than the corpus’s background noise. That is the quantitative statement of the problem, it is computable from what you already have, and it is decisive.

Check rank stability. Run the same query twice with a trivially reworded variant. If the top ten change substantially, the ordering is not resting on a real difference.

The part that isn’t a retrieval fix

Two responses lie outside this stage, and both are worth naming so the effort goes to the right place.

The distinguishing values should be extracted into fields at ingest so they can be filtered and matched structurally. And the repeated template text is boilerplate that inflates every representation toward the mean — reducing it is an ingestion decision, not a query-time one, and it is the intervention with the largest effect on this specific problem.

At the retrieval layer, the correct response to a low-contrast corpus is to stop expecting similarity to discriminate, lean on the signal that keys off rare tokens, and push everything expressible as a comparison into a predicate. Recognising which of your failures are this failure rather than a ranking failure is what stops you from spending a month tuning a merge over a list that never contained information.

densediagnosis