One question, three searches

A procurement analyst asks “which suppliers are approved for food-grade packaging and what is the audit interval for them?” Two questions in one string. Retrieval returns documents about supplier approval, documents about audits, and nothing that serves both.

This is not a ranking failure. A single retrieval pass produces a single ranked list, and there is no ordering of that list in which the top few passages cover two unrelated requirements.

Query: "which suppliers are approved for food-grade packaging and what is
        the audit interval for them?"

LEXICAL (BM25)                            DENSE (vector)
1. Supplier approval process           ~   1. Supplier qualification overview ~
2. Audit interval matrix               ~   2. Approved materials list         ~
3. Food-grade material standards       ~   3. Audit programme summary         ~
4. Packaging specification index       ✗   4. Food contact compliance         ~
5. Supplier onboarding checklist       ✗   5. Supplier audit scheduling       ~

Every result is partially relevant and none is sufficient. That signature — a page of ~ marks with no ✓ — is the fingerprint of a compound query, and it is diagnostically useful because it looks different from every other retrieval failure.

Why the average is the worst answer

Dense retrieval literally averages. One embedding represents the whole string, and a string about two topics maps to a point between them. The nearest documents to that midpoint are documents that are mildly about both — overviews, index pages, glossaries — which are the least informative documents you own. A query about A and B retrieves documents about neither, specifically.

Lexical retrieval splits its weight. Term scores accumulate, so a document strongly matching half the query competes against a document weakly matching all of it, and length normalisation decides the contest on grounds unrelated to your question. The winner is arbitrary with respect to what you asked.

Neither signal is broken and no fusion method repairs it, because both input lists are wrong in the same way. The problem is upstream of retrieval.

Splitting, and what “correctly” means

Decomposition rewrites one query into several, each retrievable on its own:

"which suppliers are approved for food-grade packaging and what is
 the audit interval for them?"

  → "suppliers approved for food-grade packaging"
  → "audit interval for approved suppliers"

Two properties matter more than the split’s elegance.

Each sub-query must stand alone. “What is the audit interval for them” carries a pronoun, and a retriever has no way to resolve it. If decomposition leaves a dangling reference, it has produced a sub-query that retrieves the wrong thing confidently — the same trap as reference resolution in a rewrite, now multiplied.

The split must be a partition, not a paraphrase. A model asked to decompose will sometimes return three restatements of the same question. You then pay three retrievals for one result set, and the merge is fed correlated lists that look like consensus. Cheap check: if the sub-queries’ result sets overlap almost entirely, the split did nothing.

Merging the lists

Three arrangements, and the choice is not cosmetic.

Union with per-sub-query quotas. Take the top few from each sub-query’s own list and concatenate. Guarantees coverage — every part of the question is represented — and guarantees mediocre ranking, because a passage’s rank within its own sub-list says nothing about its importance to the whole question.

Fuse everything. Pool all results and run reciprocal rank fusion across all the lists at once. Wrong, and worth understanding why: fusion rewards documents appearing in multiple lists, but for a decomposed query, appearing in every sub-list is a symptom of being a vague overview rather than evidence of relevance. Fusion’s consensus assumption holds across retrievers looking for the same thing; it inverts across sub-queries looking for different things.

Quota, then rerank the pool. Reserve slots per sub-query so nothing is starved, then let a cross-encoder order the combined set against the original question. The reranker sees each passage next to the full request and can judge whether it contributes, which neither the split nor the merge can. This is the arrangement that actually works, and it costs a reranking pass over a larger candidate set than a single query would have produced.

Whichever you choose, deduplicate before merging. Sub-queries overlap, the same passage arrives from two of them, and an undeduplicated final set spends two of its slots on one passage.

Costs, plainly

Retrievals multiply. Three sub-queries across two retrievers is six index queries per user question. Each is individually cheap; the total is not nothing, and it is concurrent load on the same infrastructure.

A model call precedes retrieval. Splitting reliably needs a language model, so the latency lands in front of the pipeline where nothing can hide it.

The candidate set grows, which pushes rerank cost up in proportion — the stage that was already the expensive one.

Debuggability drops. When the answer is wrong you now have to establish which sub-query failed, so log the decomposition and every sub-list, or the usual diagnostic procedure has nothing to work with.

When not to split

Single-topic queries. Splitting a query that didn’t need it produces near-duplicate sub-queries and pure overhead. Gate the stage: only decompose when a cheap test suggests compounding — a conjunction joining two clauses that each contain their own content words, more than one question mark, an enumeration.

Queries where the parts are genuinely dependent. “Which supplier had the most audit findings last year” cannot be split into “suppliers” and “audit findings” and reassembled; answering it requires aggregating across documents, and no arrangement of retrieval delivers that. Recognising this class matters: it is not a retrieval problem, and effort spent on decomposition is wasted on it.

Comparisons, sometimes. “How does the packaging standard differ between the EU and Japan” splits cleanly and usefully into two searches. “Which of our suppliers is cheapest” does not. The difference is whether each part has its own answer in a document somewhere.

Telling whether it’s your problem

Count conjunctions in your query log. A corpus of short keyword queries has no compound problem worth solving; a chat interface over a policy corpus has a large one, because people ask the question they actually have and it usually has two halves.

Then look for the fingerprint from the top of this post — result sets that are uniformly partially relevant. Sample the queries where a human reviewer marked every returned passage as “related but not answering”, and read them. If most contain two requirements, decomposition is your intervention. If most contain one requirement expressed vaguely, it isn’t, and you should be looking at which signal missed instead.