Which signal failed

A query comes back with nothing useful. Someone suggests tuning the fusion weights. Someone else suggests a different embedding model. Both are guesses, and in a hybrid pipeline there are at least five places the failure could have occurred.

The fix is different for each. So find out which one first.

The decision procedure

Take one failing query and the document that should have answered it. Walk down until something is false.

1. Is the document in the corpus?
      no  → ingestion problem. Stop.
2. Is it in the index — both indexes?
      no  → indexing problem. Stop.
3. Does the lexical retriever return it in its top 50?
4. Does the dense retriever return it in its top 50?
      both no → recall failure. Go to §recall.
5. Is it in the fused list?
      no  → fusion problem. Go to §fusion.
6. Is it in the final top-k after reranking?
      no  → ordering problem. Go to §ordering.
7. It was passed to the model and the answer was still wrong.
      → not a retrieval problem.

Most of the value is in doing this at all. Teams routinely spend a week tuning fusion for a corpus whose failing documents were never indexed.

Instrument for it. Log, per query: each retriever’s returned IDs and ranks, the fused ranks, the post-rerank ranks, and what was finally passed to the model. Without that, every step above is a manual re-run, and nobody does it more than twice.

Step 1–2: is it even there

The unglamorous checks that catch a surprising share of failures.

Missing from the corpus. The document exists in the source system and never made it through ingestion — an unsupported file type, a parse failure swallowed by a try/except, a crawler that stopped at a pagination boundary. Check whether ingestion reports per-document success, and whether anyone reads it.

Present in one index only. Two indexes built by two pipelines drift. A document added to the vector store during a backfill that never touched the lexical index will be invisible to lexical search forever, and the symptom looks exactly like a lexical retrieval failure.

Present but empty or mangled. The document was ingested and its text extraction produced whitespace, or navigation chrome, or the first page of a scanned PDF. Look at the stored text, not at the fact that a record exists.

Filtered out. A date, tenant, or type filter is excluding it. Re-run the query with filters off. This is fast, and it explains failures that otherwise look inexplicable — particularly when a chunk is missing the metadata field your filter tests, since a null usually fails the predicate silently.

Chunked into uselessness. The document is indexed, but the answer got split across two chunks and neither one contains it whole. That’s an upstream problem and not fixable here; recognising it is the job.

§recall: neither retriever found it

Both retrievers missed. Look at how they missed, because the query classes are well-defined.

Is the query identifier-shaped? A SKU, error code, version, path, function name. Dense retrieval is expected to fail; lexical should have caught it. If it didn’t, suspect the analyser — a tokeniser splitting ERR_2041 into err and 2041 destroys the rare-term advantage that made it findable. Check what your analyser actually produces for that string. This is the single most common fixable recall failure.

Is it a paraphrase? No content-word overlap with the document. Lexical is expected to fail; dense should have caught it. If it didn’t, the passage may be too long — a chunk covering many topics has a diluted vector — or the domain vocabulary may be far enough from the embedding model’s training distribution that its representations aren’t discriminating well.

Does it contain negation, or a numeric constraint? Neither retriever handles these. “Not X”, “under 500 MB”, “released after 2024”. The fix is a query rewrite or a metadata filter, before retrieval. No amount of tuning helps.

Is it multi-hop? The answer requires combining two documents, neither of which is individually responsive. Retrieval isn’t the failing component; the pipeline needs decomposition, or the question is out of scope.

Is the document simply not distinctive? Some documents are near-duplicates of a dozen others. Retrieval returns one of them, arbitrarily. This is a corpus problem — deduplicate.

§fusion: found, then lost in the merge

Both retrievers had it, and the merged list doesn’t. Rarer, and specific.

Was it found by only one retriever, at a mediocre rank? RRF rewards agreement. A document at rank 20 in one list with no support from the other scores poorly by design. This is usually correct behaviour, and it’s wrong for exactly the query classes where one signal is supposed to be the only one that works — identifier lookups. Consider routing those queries to lexical alone rather than fusing.

Did you retrieve enough candidates? Fusion can only rank what it was given. If each retriever returns 10 and the answer is at rank 12 in one of them, it never entered the merge. Widening the per-retriever k is the cheapest fix available and fixes more than fusion tuning does.

Are the document IDs consistent? If the two retrievers identify the same document differently, the consensus bonus never fires and every result looks single-sourced. Check by counting how many documents appear in both lists for a typical query — if it’s near zero, your IDs don’t match.

Are the filters consistent? A filter applied to one retriever and not the other silently halves your candidates.

§ordering: found, ranked too low

It’s in the fused list at rank 12, and you pass three passages to the model.

This is what reranking is for, and the check is direct: does a cross-encoder promote it? If yes, you have an ordering problem with a known solution. If no, the passage may not actually answer the query as well as you think — worth re-reading before concluding the pipeline is wrong.

Also worth checking: is your final top-k too small? Passing three passages when five would fit is a self-inflicted ordering problem. How many you can afford is a window-budget question, not a retrieval one, but the interaction is real.

Doing this at scale

One query is a diagnosis. A pattern is a fix.

Take fifty to a hundred failing queries and run the procedure on each, recording only the step at which it failed. The distribution tells you where to work:

Failed at step 1–2 (not indexed / filtered)     18
Failed at recall (neither retriever)            31
Failed at fusion                                 4
Failed at ordering                              22
Not a retrieval failure                         25

(Illustrative counts — yours will look nothing like this.)

A distribution like that says: stop tuning fusion. Two-thirds of the effort belongs in recall and ordering, and a fifth of the “retrieval failures” aren’t retrieval failures at all.

Then subdivide the largest bucket. Of the recall failures, how many were identifier-shaped? If most, your analyser configuration is one afternoon’s work away from a large improvement. That’s a much more useful conclusion than “hybrid search needs tuning.”

The habit worth building

Never change a retrieval parameter without knowing which step it addresses.

Fusion weights fix fusion problems. A different embedding model fixes a narrow subset of dense recall problems. A reranker fixes ordering problems. Each is useless against the other categories, and each carries a cost — latency, index size, re-embedding the corpus — that you’ll have paid for nothing.

The procedure above takes a few minutes per query once the logging exists. The logging is the actual investment, and it’s the difference between debugging this system and guessing at it.