Fusing two ranked lists

You’re running both retrievers. Each returns ten results with a score attached. Now you need one list of ten.

The obvious move — add the scores — is wrong, and it’s wrong in a way that produces a working system with quietly bad ranking, which is the worst kind of wrong.

Why you can’t add the scores

A BM25 score and a cosine similarity are not the same kind of number.

Cosine similarity is bounded. It falls in a fixed range determined by the geometry, and in practice, on a normalised embedding space, relevant results tend to cluster in a narrow band near the top of that range. The difference between an excellent match and a mediocre one can be small in absolute terms.

BM25 is unbounded and corpus-dependent. There is no maximum. A score depends on term rarity in your corpus, document length, and the scoring parameters. The same document, same query, in a corpus twice the size, scores differently. A score of 12 might be exceptional for one query and unremarkable for another, because it depends on how rare the query’s terms happen to be.

So:

Query: "ERR_2041 signature failure"

LEXICAL                          score    DENSE                          score
1. ERR_2041: signature failed    18.4     1. Certificate validation      0.89
2. Signature verification guide   9.1     2. Signature verification      0.87
3. Error code reference           4.2     3. ERR_2041: signature failed  0.86

Add the raw scores and the lexical results dominate completely — 18.4 swamps 0.89, and the dense retriever might as well not be running. Multiply the dense scores by 20 to compensate and you’ve tuned for this query; the next one, with common terms, has a top BM25 score of 3.

Raw scores from different retrievers are not comparable, and no fixed multiplier makes them comparable. That’s the constraint every fusion method works around.

Reciprocal rank fusion

The standard answer, and the one to start with. RRF ignores scores entirely and uses only rank.

For each document, sum across the lists it appears in:

score(d) = Σ  1 / (k + rank(d, list))

where rank is 1-based and k is a constant — 60 is the value most implementations start from, and it’s a starting point to tune, not a magic number.

Applied to the lists above (k = 60):

ERR_2041: signature failed
    lexical rank 1 → 1/61 = 0.0164
    dense   rank 3 → 1/63 = 0.0159
    total = 0.0323                       ← rank 1

Signature verification guide
    lexical rank 2 → 1/62 = 0.0161
    dense   rank 2 → 1/62 = 0.0161
    total = 0.0322                       ← rank 2

Certificate validation
    dense   rank 1 → 1/61 = 0.0164
    total = 0.0164                       ← rank 3

The document both retrievers liked wins. The one only dense retrieval found still places, but below anything with agreement.

What k does. It flattens the curve. Small k makes the top rank dramatically more valuable than the second — near-winner-takes-all. Large k compresses the differences, so appearing in both lists matters more than placing highly in either. The default sits toward the compressed end, which is usually what you want from a fusion method: it rewards agreement.

Why it works well:

No normalisation, no scale problem, nothing to calibrate per corpus.
Robust. A retriever returning wild scores can’t hijack the merge, because its scores are discarded.
Trivially extends to three or more retrievers.
Naturally rewards consensus, which is a reasonable prior when neither signal is trustworthy alone.

What it throws away — and this is a real cost, not a footnote:

RRF cannot tell the difference between a retriever’s rank-1 result being a perfect match and its rank-1 result being the least bad of a bad set. Both are “rank 1”. When a query matches nothing in the corpus, every retriever still returns something in rank order, and RRF confidently fuses garbage.

It also can’t express that one result is far better than the runner-up. Ranks 1 and 2 are always one step apart, whether the gap in actual relevance was enormous or negligible.

Consequence: if you need a relevance threshold — “return nothing rather than something irrelevant” — RRF gives you no basis for one. You have to get that from the underlying scores before fusion, or from a reranker afterwards.

Score normalisation and weighted sums

The alternative: map both score sets onto a comparable scale, then combine with weights.

normalised = (score − min) / (max − min)      # min-max, per query
combined   = w · dense_norm + (1 − w) · lexical_norm

What it buys. Score magnitudes survive, so a document that dominated one retriever keeps that advantage. And w is an explicit tuning knob — you can weight toward lexical for an identifier-heavy corpus, toward dense for a conversational one.

Where it breaks:

Min-max normalisation is per-query and relative. The top result always normalises to 1.0, even when it’s terrible. You’ve mapped “best of a bad list” onto the same value as “perfect match”, which throws away exactly the information the method was supposed to preserve.

It’s sensitive to outliers. One anomalously high BM25 score compresses everything else toward zero.

The lists have different lengths and different members. Documents found by one retriever and not the other need a score for the missing side. Zero is the usual choice and it’s a strong, arbitrary penalty.

w needs tuning per corpus, and it drifts as the corpus and query mix change. RRF’s k also tunes, but it’s far less sensitive.

Variants exist — z-score normalisation instead of min-max, or normalising against a fixed reference distribution rather than per-query — and they mitigate some of this at the cost of more machinery.

Choosing

	RRF	Weighted normalised sum
Calibration needed	Almost none	Per corpus, ongoing
Uses score magnitude	No	Yes
Robust to a misbehaving retriever	Yes	No
Supports a relevance threshold	No	Weakly
Expresses “prefer lexical here”	No	Yes
Extends to N retrievers	Trivially	Awkwardly

Start with RRF. It works acceptably out of the box, it has one insensitive parameter, and it can’t be broken by a scale change in one retriever. The large majority of hybrid systems need nothing more.

Move to weighted normalisation when you have a measured reason to favour one signal, and an evaluation set to tune the weight against. Not before — an untuned weight is worse than no weight.

Two things that are not fusion strategies, and are sometimes better than either:

Route instead of fusing. Classify the query and send it to one retriever. Identifier-shaped queries to lexical, conversational ones to dense. Cheaper than running both, and it fails hard when the classifier is wrong.

Fuse, then rerank. Use RRF purely as a candidate generator — take the union of both lists, merged crudely — and let a cross-encoder do the actual ordering. In this arrangement fusion quality barely matters, because you only need the right document somewhere in the candidate set. This is the design most high-quality pipelines converge on.

Practical notes

Retrieve more from each retriever than you intend to keep. If you want ten final results, take thirty or fifty from each. Fusion can only rank what it was given, and a document sitting at rank 15 in both lists is exactly the kind of consensus result you want to surface.

Deduplicate before fusing. The two retrievers must agree on document identity, or the same document appears twice under different IDs and the consensus bonus never fires. Sounds obvious; breaks constantly when the two indexes were built by separate pipelines.

Log both source ranks on every fused result. When a query goes wrong, you need to know whether a document was missing from both lists or merely ranked badly in the merge — different failures with different fixes.

Apply filters consistently to both retrievers. A date or tenant filter applied to one and not the other produces a merged list where half the results violate the constraint.