Searching with a hypothetical answer

A question and its answer are written differently. “Why does the boiler lock out on a cold morning?” looks nothing like the paragraph in a service manual that explains condensate freezing, and a dense retriever comparing the two is comparing a question-shaped vector against an explanation-shaped one.

One response to that: don’t search with the question. Ask a model to write the answer it would guess at, then search with the guess.

Query: "why does the boiler lock out on a cold morning?"

DENSE, question embedded              DENSE, hypothetical answer embedded
1. FAQ: common lockout questions  ~   1. Condensate pipe freezing        ✓
2. Troubleshooting flowchart      ~   2. Cold-weather lockout codes      ✓
3. Cold weather operating notes   ~   3. Trap and drain routing          ~
4. Why boilers short-cycle        ✗   4. FAQ: common lockout questions   ~
5. Annual service checklist       ✗   5. Frost protection settings       ~

The hypothetical answer contained words like condensate, freezing, drain, and trap — none of which were in the question, all of which are in the document. The generated text was probably wrong in its specifics. It didn’t need to be right; it needed to be in the right neighbourhood.

The mechanism, stated precisely

Dense retrieval scores similarity between two embeddings. An embedding model places text according to what the text is about and also, unavoidably, according to what kind of text it is. Questions cluster with questions. This is why FAQ pages surface disproportionately for question-shaped queries in the left-hand list above — they are the corpus’s question-shaped documents, and shape is part of the similarity.

Generating a hypothetical answer converts the query into the same register as the corpus. You are not improving the query’s meaning; you are moving it into the region of the space where the answers live.

That framing predicts the technique’s boundaries better than “the model adds knowledge” does. It explains why a factually wrong hypothetical still works, why it helps least on corpora that are themselves question-shaped, and why it does approximately nothing for the lexical signal beyond supplying additional terms — an effect closer to expansion, with expansion’s drift risk attached.

Where it earns its cost

Short, jargon-free questions over a technical corpus. The bigger the register gap between how users ask and how documents explain, the more there is to gain.

Domains with a vocabulary the user doesn’t have. A patient describing symptoms in plain language, searching a corpus written in clinical terms. The generated answer supplies the intermediate vocabulary, and this is the case where the technique is most clearly better than a synonym list, because the mapping is compositional rather than term-by-term.

Zero-shot deployments with no query log. Before you have traffic, you have no data to mine for synonyms and no labels to tune fusion weights against. A generative rewrite needs neither.

The three ways it misfires

Confident hallucination toward a real but wrong region. The generated answer invents a mechanism that is plausible and absent from your corpus — but similar to a different topic you do document. The search then retrieves that topic, excellently. This is worse than a bad ranking: the results are coherent, on a single subject, and about the wrong thing. Sparse, unfamiliar corpora are most exposed, because the model has the least basis for a good guess exactly where you most need one.

Identifier and entity queries get destroyed. Asked to elaborate on SKU 4471-B, a model produces a paragraph of generic product prose. The one discriminating token in the original query is now a small fraction of a longer text, and the embedding of that text is near every product description you own. The identifier failure mode is not merely unfixed; it is amplified. Detect identifier-shaped queries and skip this stage entirely.

Latency in front of everything. A generation call precedes retrieval, so its cost is added before the pipeline starts and cannot be overlapped with anything. In a pipeline that also reranks, you are paying for two model passes around a retrieval step. Whether that is acceptable depends on your budget; what is not acceptable is discovering it after deployment.

There is a fourth, quieter problem: non-determinism. The same question produces different hypothetical answers on different runs, so the same question produces different result sets. That complicates every kind of comparison you might want to run, and it means a user who retries a query gets different sources with no explanation.

Variants that mitigate some of it

Generate several, embed each, fuse the result lists. Multiple hypotheticals cover more of the answer space and a wrong one is outvoted. Costs more generation and a merge whose inputs are highly correlated, so consensus in the fusion means less than it does when merging genuinely different signals.

Search with both the question and the hypothetical, then fuse. The hedge. If the generation went wrong, the question’s own list is still there. This is the variant most worth defaulting to, because its failure mode is “no improvement” rather than “confidently wrong”.

Keep the question’s terms in the generated text. Prompting the model to include the query’s key terms verbatim anchors the result and limits drift.

Cache by normalised query. Common questions recur, and a cache removes both the latency and the non-determinism for the queries users actually repeat.

What it does not fix

It is not a substitute for the lexical signal. Nothing about generating an answer helps you find an error code or an exact phrase. The technique addresses vocabulary and register, which is the dense retriever’s weak flank in the direction it was already strong.

It does not add knowledge to your corpus. If the answer isn’t indexed, a better query finds a better-sounding wrong document. Some of the enthusiasm around this trick is really enthusiasm about generated text appearing in the pipeline; the retrieval still only returns what you have.

It does not fix ranking within a good candidate set. That is the reranker’s job, and a cross-encoder reading the original question against each candidate is a more direct attack on the same register gap — it compares question to passage with full attention across both, rather than approximating the comparison in advance.

Deciding whether to try it

The test is cheap and worth running before committing to the latency.

Take fifty question-shaped queries with known correct passages. Record the rank of the correct passage under the plain question. Then generate a hypothetical answer for each, retrieve again, and record the rank. You are looking for the shape of the change, not an average: if a subset of queries improves dramatically and the rest are unchanged, identify what that subset has in common, and consider applying the technique only to queries matching that description. If a subset gets dramatically worse, read those generated answers — they will show you the hallucination-toward-a-real-region failure directly.

How large the effect is depends entirely on the register gap between your queries and your corpus, which is a property of your users and your documents and not something anyone can tell you in advance.