Weighting one signal over the other

Someone on the team notices that dense retrieval is doing most of the useful work and proposes weighting it higher in the merge. It is a reasonable proposal, it will improve the average, and it will make one class of query permanently worse.

That trade is the whole subject. A single weight applied to every query is an assertion that your queries are all the same kind, and they aren’t.

Query A: "how do I request a hardware exception"

LEXICAL (BM25)                            DENSE (vector)
1. Exception request workflow          ✓   1. Exception request workflow      ✓
2. Hardware standards policy           ~   2. Requesting a policy waiver      ✓
3. Exception reporting in finance      ✗   3. Hardware standards policy       ~
4. Requesting equipment               ~   4. Approval routing for exceptions  ~
5. Hardware disposal procedure        ✗   5. Requesting equipment            ~

Query B: "TX-9910 firmware exception"

LEXICAL (BM25)                            DENSE (vector)
1. TX-9910 firmware release notes      ✓   1. Firmware update procedure       ~
2. TX-9910 known issues                ✓   2. TX-9800 firmware notes          ✗
3. Firmware exception codes             ~   3. Firmware exception codes        ~
4. TX-9800 firmware notes              ✗   4. Device certification list       ✗
5. Firmware update procedure           ~   5. TX-9910 release notes           ✓

Weight dense higher and query A improves marginally while query B loses its only good list. The average across a query set dominated by A-shaped queries goes up, and the B-shaped queries — the ones users complain about — get worse.

What a weight actually multiplies

Two fusion families and the weight means something different in each.

In a weighted normalised sum, the weight multiplies a normalised score, so it directly trades one retriever’s magnitudes against the other’s. In reciprocal rank fusion, scores are discarded and a weight multiplies a rank contribution instead:

score(d) = Σ  w_list / (k + rank(d, list))

The RRF form is better behaved, because it cannot be destabilised by a retriever whose scores drift — the reason RRF is the default. It is also less expressive: it can say “prefer the lexical list” but not “prefer this lexical result, which scored extraordinarily”.

Both share the important property: the weight is global across queries unless you deliberately make it otherwise. Every query pays for it.

Deriving a weight instead of guessing one

An untuned weight is worse than no weight, because the default of equal weighting is at least unbiased. So the sequence matters.

First, a labelled query set. Fifty to a few hundred real queries with a known correct passage each. Drawn from your query log, not invented — an invented set encodes your assumptions about the query mix, which is the exact thing you are trying to measure.

Second, sweep the weight coarsely. Compute your ranking measure at a handful of weights across the range. You are looking for the shape of the curve, not its maximum. A flat curve means the weight doesn’t matter and you should stop. A sharp peak on a set of fifty queries usually means you have fitted to a handful of them.

Third, and this is the step that gets skipped: break the result down by query class. Split your set into identifier-shaped, paraphrase-shaped, and everything else, and sweep within each. The near-certain result is that the classes want opposite weights. That finding is more useful than any single number, because it tells you the problem is not the weight’s value.

Fourth, re-derive it on a schedule. A weight is fitted to a query distribution and distributions drift as users discover what your system answers well. A weight tuned once and never revisited is a constant fitted to a year-old population.

Per-query weighting, which is really routing

Once you have established that different query classes want different weights, the honest conclusion is that the weight should vary per query. That is a query classifier, and at that point you are routing with soft edges rather than hard ones.

Which is often the right answer, and comes with the classifier’s failure mode: a misclassified query is now weighted against the signal that would have found it. Soft weighting is more forgiving than hard routing here — a misclassified query with a bad weight still has the other list present at reduced influence, where a misrouted query has no other list at all. If you are unsure of your classifier, weight; if you are confident, route.

A cheap version that avoids a trained classifier entirely: derive the weight from surface features of the query. Short queries and queries containing tokens that look like identifiers lean lexical; long natural-language queries lean dense. A handful of rules, inspectable, no training data, and it captures most of the variance because most of the variance is that one distinction.

When the weight is the wrong knob

Three situations where tuning it is a distraction, and all three are common enough to check first.

The candidate lists are too shallow. If the correct passage was never in either list, fusion weighting is rearranging a set that doesn’t contain the answer. Depth is the cheaper fix and it is frequently mistaken for a fusion problem.

A reranker follows the fusion. If a cross-encoder reorders the merged candidates, fusion’s job is reduced to getting the right document into the candidate set, and its ordering barely matters. Tuning a weight to improve an ordering that is about to be discarded is pure waste. In this arrangement, prefer whatever weighting maximises coverage of the candidate set — usually equal weights, since each signal contributes documents the other lacks.

One retriever is misconfigured. A weight of zero on the lexical list and an analyser that fragments your identifiers look identical from the outside: the lexical list is useless. One of those is fixed in an afternoon and the other is a permanent concession. Before down-weighting a signal, establish that the signal is working.

The rule this site keeps returning to

A weight that eliminates one signal is not a tuning result; it is a decision to run one retriever.

If your sweep says the optimal weight puts nearly all the mass on one side, take the conclusion seriously in both directions: either your query mix genuinely only needs one signal — in which case stop paying for the other, including its index and its infrastructure — or your labelled set doesn’t represent your query mix. Both are actionable. Deploying an extreme weight while continuing to maintain the suppressed retriever is the one outcome that costs money for nothing.

And keep the per-class breakdown where the team can see it. The reason this argument recurs on every retrieval project is that the aggregate number always supports whoever is speaking, and the breakdown is the only thing that settles it.