Two Signals

How many candidates to retrieve

An insurance analyst asks “what happens to a claim if the adjuster’s report is late”. The passage that answers it is indexed, both retrievers rank it, and the answer is still wrong — because each retriever was asked for ten results and the passage sits at rank fourteen in one and rank twenty-two in the other.

Nothing in that pipeline is misconfigured. Four separate cut-offs are, and only the first one had a chance to matter.

Query: "what happens to a claim if the adjuster's report is late"

LEXICAL (BM25)                            DENSE (vector)
1.  Adjuster assignment SLA           ~   1.  Claim lifecycle overview        ~
2.  Late payment penalties            ✗   2.  Adjuster responsibilities       ~
3.  Report templates for adjusters    ✗   3.  Delays and claimant notice     ~
4.  Claim status codes                ✗   4.  Reopening a closed claim        ✗
5.  Adjuster licensing by state       ✗   5.  Escalation paths               ~
    …                                     …
22. Missed-deadline handling          ✓   14. Missed-deadline handling        ✓

Four cut-offs, not one

Every hybrid pipeline narrows a candidate set four times, and each stage can only work with what the previous one handed it.

corpus  ──lexical k────┐
                       ├── fuse → k_f ── rerank → k_r ── prompt → k_p
corpus  ──dense k──────┘
  • Retrieval depth — how many results each retriever returns. The only stage that can add a document to the pipeline.
  • Fusion depth — how many merged results survive the merge.
  • Rerank depth — how many candidates the cross-encoder scores.
  • Final depth — how many passages reach the model.

Only the first cut-off decides which documents exist. The other three only reorder and discard. That asymmetry is the whole lesson: a document lost at retrieval depth is unrecoverable by any amount of reranking, while a document ranked badly at fusion depth is recoverable for the price of a scoring pass.

Why retrieval depth is usually set too low

The default in most client libraries is a small number, chosen so that a getting-started example prints a readable amount of output. It survives into production because the demo queries work — demo queries are the ones where both retrievers put the answer near the top.

The queries that need depth are the ones where one signal is confidently wrong. Lexical retrieval promotes documents dense with the query’s terms; on a corpus that discusses adjusters constantly, “adjuster” and “report” are common tokens and the ranking is driven by whichever document repeats them most, not the one that answers the question. Dense retrieval promotes documents about the same topic; a corpus of claims procedures is all about the same topic. Both retrievers are working. Both need twenty results before the specific passage surfaces.

Worse, this failure is invisible in aggregate. The system answers most questions, and the ones it misses look like the corpus is missing content — which is a different failure with a different fix.

What extra depth costs at each stage

Retrieval depth is cheap. Both retrievers are already traversing their index; returning fifty instead of ten mostly changes how many results are serialised back. The cost grows with the size of each returned record, so return identifiers and scores at this stage, not full document text, and hydrate the survivors later.

Fusion depth is nearly free. Fusion is arithmetic over a few dozen entries.

Rerank depth is the expensive one. A cross-encoder runs a model over every query-passage pair, so its cost is linear in the number of candidates and there is no batching trick that removes it. Doubling rerank depth roughly doubles that stage’s latency. This is where a budget actually binds, and it is why the reranker’s position in the pipeline matters more than its accuracy.

Final depth is not a retrieval decision. How many passages fit is a window and cost question belonging elsewhere; retrieval’s obligation is to make the right passage available within that number.

The useful consequence: be generous where it’s cheap and strict where it’s expensive. A pipeline that retrieves deeply, fuses everything, reranks a moderate slice, and passes a handful is spending its money in the right order. A pipeline that retrieves ten and reranks all ten has inverted it — it pays for a cross-encoder to reorder a candidate set that probably never contained the answer.

Finding the depth your corpus needs

Do not pick a number. Measure where the answer actually sits.

Take fifty queries with a known correct passage. For each, run both retrievers at an absurd depth — several hundred — and record the rank at which the correct passage appears in each list. You now have two distributions, and they answer the question directly: the depth you need is the one that covers most of your distribution, and the shape tells you whether depth is your problem at all.

Three shapes and what each means:

  • Both distributions concentrated in the top few. Your retrieval depth is fine. Spend the effort on ordering.
  • One distribution has a long tail. That signal needs more depth than the other. Which one it is tells you something about your query mix — a lexical tail suggests paraphrase-heavy queries, a dense tail suggests identifier-shaped ones.
  • The passage is absent from both lists at any depth. Depth is not your problem. That’s a recall failure or an indexing failure.

Re-run this when the corpus grows substantially. A larger corpus pushes every relevant document further down both lists simply because there is more competition, so a depth that was adequate at ten thousand documents can be inadequate at a million, with no code change and no visible cause.

Asymmetric depth is allowed

There is no rule that both retrievers return the same number, and no reason they should. If your measurement shows the lexical tail is long and the dense one isn’t, take more from lexical.

Two caveats. First, asymmetric depth interacts with fusion: reciprocal rank fusion rewards agreement, and a document that only appears in the deeper list has no partner to agree with, so it competes on one weak contribution alone. Second, the imbalance is a tuning decision that ages — it was derived from a query mix, and query mixes drift as users learn what your system can do.

The failure this prevents

The specific bug worth internalising: a document that both retrievers ranked, that fusion would have promoted for exactly the consensus reason fusion exists, never entering the merge because both lists were truncated above it. The two signals agreed and the pipeline never found out.

That is the cheapest fixable failure in a hybrid system, it costs one configuration value, and it is routinely diagnosed as an embedding-model problem instead. Before changing a model, changing a fusion weight, or adding a reranker, ask both retrievers for more and see whether the problem was arithmetic the whole time.

How much more is corpus-dependent — a homogeneous corpus of near-identical procedural documents needs far more depth than a diverse one, because every document is a plausible neighbour of every query. Measure it rather than inheriting a default.

tradeoffsfusion