When five results say the same thing

A bank’s internal search is asked “what identification is acceptable for a joint account”. Five passages come back, all relevant, all from the same onboarding manual, all restating the same list of documents. The passage that covers the additional requirement for one account holder being a non-resident is at rank nine and never makes it into the answer.

Nothing here is a relevance failure. Every returned passage deserved to be returned. The problem is that the top five contain one fact between them.

Query: "what identification is acceptable for a joint account"

FUSED TOP FIVE
1. Onboarding manual §4.1 — accepted ID types        ✓
2. Onboarding manual §4.1 — accepted ID types (rev)  ✓ (duplicate content)
3. Onboarding manual §4.2 — ID verification steps    ✓
4. Branch quick guide — accepted ID types            ✓ (restatement)
5. Onboarding manual §4.1, older revision            ✓ (superseded)
   …
9. Joint accounts with a non-resident holder         ✓ (the missing piece)

Relevance is not coverage

A ranked list is produced by scoring each document against the query independently. Nothing in either retriever, and nothing in fusion, considers what the other selected results already contain.

Both retrievers are optimising the wrong objective for a top-k answer. They are answering “which documents are most relevant” when what a downstream answer needs is “which small set of documents covers the question”. Those coincide when a corpus has one passage per fact. They diverge sharply when it doesn’t, and most real corpora don’t — policies get restated in summaries, product details appear in a spec sheet and a datasheet and a support article, revisions accumulate.

Worse, the two signals both amplify it. Lexical retrieval scores near-identical texts near-identically, so duplicates cluster. Dense retrieval places near-identical texts at near-identical points, so duplicates cluster. And fusion rewards agreement, so a passage duplicated across the corpus and retrieved by both signals rises above a unique passage found by one.

Where the duplication comes from

Three sources, and only one of them is a query-time problem.

The corpus genuinely contains repetition. Restatements, summaries, translations, revisions. Real content, legitimately indexed, and the duplication is a fact about your documents.

The same document was indexed more than once. Two ingestion paths, a re-crawl that didn’t deduplicate, a migration that ran twice. This is an ingestion defect and the right fix is upstream — a query-time cap papers over it while your index size and costs carry the duplication forever. Diagnose it by checking whether the duplicate results have distinct identifiers with identical text.

One document contributed several adjacent passages. Common and often correct — adjacent passages from the same section really are the most relevant material. It becomes a problem when they crowd out everything else, and it is a query-time concern because the fix is a selection rule, not a re-ingestion.

Only the third, and the first, belong to this stage. If your duplicates are the second kind, stop here and fix the index.

Three query-time remedies

A per-source cap. The blunt instrument: allow at most a fixed number of passages from any one document, or one section, or one source system. Cheap, deterministic, easy to explain, and it takes one line in the selection step after fusion.

Its cost is that it is wrong whenever the answer genuinely lives in one document. A question about a single procedure is best answered by four consecutive passages from that procedure, and a cap of one truncates it. Set the cap loosely — the goal is preventing a monoculture, not enforcing variety.

Near-duplicate collapse. Detect that two candidates say the same thing and keep one, preferring the authoritative or most recent. Text-level fingerprinting catches exact and near-exact restatements cheaply. Embedding-level similarity between candidates catches paraphrases and costs a small number of comparisons among the candidate set, which is affordable because the set is small.

The cost is a threshold, and thresholds on similarity do not travel — a collapse threshold tuned on one corpus will over-merge on another. Prefer collapsing on text fingerprints, where the notion of “the same” is much better defined, and use embedding similarity only where you have verified the threshold.

Maximal marginal relevance. Select results one at a time, each time picking the candidate that maximises a mix of relevance to the query and dissimilarity from what you’ve already picked. A single parameter trades the two.

More principled than a cap, and it has two real costs. It needs candidate-to-candidate similarity, which means either embeddings for every candidate or an extra computation over the set. And the parameter is a genuine tuning burden: pushed toward diversity, it promotes marginally relevant passages purely for being different, which is a distinct and less obvious failure than the one you started with.

Ordering with the reranker

If a cross-encoder is already in the pipeline, note what it does and doesn’t do here. A reranker scores each passage against the query, independently. A reranker will happily rank five near-identical passages as the top five, and it will be right each time — its objective is pairwise relevance, not set coverage.

So diversity is not a reranking problem and adding a reranker does not address it. The correct arrangement is: retrieve, fuse, rerank for relevance, then apply the diversity selection to the reranked candidates. Doing it in the other order throws away candidates before the stage best able to judge them, and doing it not at all leaves the duplication in place.

The counterargument worth taking seriously

Diversity can hurt. If the query has one answer and your corpus states it consistently, then five passages saying the same thing are five pieces of evidence for the same correct answer, and forcing variety into that set replaces confirmation with noise. Downstream, a set of passages that disagree produces worse answers than a set that agrees.

So this is a per-query trade, and the useful distinction is whether the question is narrow — one fact, one procedure, one value — or compound and multi-faceted, where the answer needs several different things. The second class is where duplication actually hurts, and it overlaps heavily with the queries that also want decomposition. If you already detect compound queries, that detector is a reasonable gate for applying diversity.

Telling whether it’s your problem

The measurement is direct and needs no labels: for a sample of queries, count how many distinct documents appear in the final top-k, and count how many distinct top-k passages a reviewer would call redundant.

If your top five routinely come from one or two documents, you have a monoculture. Whether it matters depends on the questions, so pair the count with the failure log: look for cases where the correct passage was in the candidate set, ranked below the cut, and the passages above it were redundant with each other. That specific pattern — the answer displaced by repetition rather than by irrelevance — is what diversity fixes, and it is a different failure from every other reason a passage doesn’t come back.