Rewriting the query before you search

A user types “hey so I think our exports have been failing since the upgrade last week, any idea why?” into an internal documentation search. Both retrievers return material about upgrades, and nothing about export failures.

The query contains four content words that matter and a dozen that actively mislead. Rewriting it before retrieval is usually cheaper and more reliable than trying to make the retrievers tolerate it.

Query: "hey so I think our exports have been failing since the upgrade
        last week, any idea why?"

LEXICAL (BM25)                            DENSE (vector)
1. Upgrade notes 4.3                  ✗   1. Upgrade planning guide          ✗
2. Weekly release schedule            ✗   2. Release cadence FAQ             ✗
3. Upgrade prerequisites              ✗   3. What changed in 4.3             ~
4. Failing health checks              ~   4. Troubleshooting after upgrade   ~
5. Export formats reference           ~   5. Common upgrade questions        ✗

Rewritten to export failure after upgrade:

LEXICAL (BM25)                            DENSE (vector)
1. Export job failures                ✓   1. Export failures after upgrade   ✓
2. Export failures after upgrade      ✓   2. Export job failures             ✓
3. Diagnosing stuck exports           ~   3. Diagnosing stuck exports        ~
4. Upgrade notes 4.3                  ~   4. Export formats reference        ~
5. Export formats reference           ~   5. Upgrade notes 4.3               ~

Why the raw query hurts both signals, differently

Lexical retrieval is diluted by filler. Term scoring weights rare terms heavily, and conversational padding is made of common ones — but “upgrade” and “week” are also common in a corpus about software releases, so they contribute little discriminating power while still shaping the ranking. The rare, decisive term here is “export”, and it is one token out of twenty-two. Length normalisation compares the query’s terms against document length, not against how much of the query was signal.

Dense retrieval is diluted by averaging. An embedding of a long, multi-topic string lands somewhere between its topics. This query is about exports, upgrades, timing, and asking for help; the resulting vector is near documents that are generally about upgrades and vaguely troubleshooting-shaped, which is exactly what came back. Dense retrieval didn’t fail to understand the sentence — it represented all of it, including the parts you didn’t want.

So the two signals are hurt by the same input for unrelated reasons, and one rewrite fixes both. That’s unusual enough to be worth exploiting: most interventions help one signal at the other’s expense.

The transformations worth doing

Ordered roughly by return on effort.

Strip conversational framing. Greetings, hedges, “any idea why”, “I was wondering if”. Deterministic and safe — a phrase list gets most of it.

Resolve references. “That error”, “the same problem”, “it” — anything pointing at something outside the query string. If the reference cannot be resolved, the query is not answerable by retrieval and should be sent back for clarification rather than searched.

Expand in-house abbreviations. Organisations run on acronyms that mean something different elsewhere, and the embedding model learned the elsewhere meaning. A maintained mapping from your internal short forms to their expansions is one of the highest-value transformations available, because it fixes a class of query where dense retrieval is not merely unhelpful but confidently wrong.

Extract structured constraints. Dates, versions, statuses, numeric comparisons, owners. “Since the upgrade last week” is a date range, and a range is a predicate, not a search term. Pulling it out both improves ranking and makes the constraint actually enforced instead of merely mentioned.

Normalise the surface form of identifiers. If your corpus writes ERR-2041 and users type err 2041, canonicalise before searching. This is a rewrite, not an analyser change, and it is worth doing at the query layer when the corpus is inconsistent enough that no analyser configuration covers it.

Split compound questions. Two questions in one string produce a vector between them and a term set belonging to neither. That deserves its own treatment.

What not to strip

Negation. “Exports that do not require a schema” — removing “not” inverts the request, and neither retriever handles negation anyway, so the correct move is to detect it and change strategy rather than delete the word and hope.

Quoted spans. A user quoting a phrase is asking for exact matching. Preserve the quotes and route that span to the lexical side intact.

Domain terms you don’t recognise. A rewriter that drops unfamiliar tokens will drop exactly the rare identifiers that give lexical retrieval its advantage. Unrecognised is not the same as noise.

Anything that changes the question. The failure mode of aggressive rewriting is a query that retrieves excellent results for a question the user didn’t ask. That failure is worse than a bad ranking, because the answer looks confident and cites real sources.

Rules or a model

Rule-based rewriting — phrase lists, abbreviation maps, regexes for identifiers and dates — is cheap, deterministic, testable, and adds effectively nothing to latency. It covers the framing, the abbreviations, and the identifier normalisation.

Model-based rewriting handles reference resolution and paraphrase, and buys that with a language-model call in front of every search. The costs are real and worth stating plainly: an added round trip before retrieval has even started, non-determinism that makes the same query behave differently on two days, a new failure mode where the rewrite is wrong, and a debugging surface where the query you logged is not the query that ran.

The middle path most systems land on: rules always, a model only when a cheap test says the raw query is unsuitable — it is long, it contains an unresolved pronoun, it ends in a question mark and has more than some number of filler tokens. Most queries then skip the model entirely.

Keep the original

Whatever you do, retrieve on the rewrite and keep the original for everything else. Log both. The original is the only record of what the user asked, it is what you show them in the interface, and it is the first thing you need when a result set is inexplicable.

A rewrite pipeline without that logging is the hardest kind of retrieval bug to diagnose, because the usual procedure starts by re-running the query — and you no longer know what ran.

It is also worth retrieving on both and fusing, when latency allows. The rewrite is a hypothesis about what the user meant; the original is the evidence. Merging the two result lists hedges the case where the rewrite was wrong, at the cost of another retrieval round and a merge whose inputs are correlated enough that consensus means less than usual.

Telling whether it’s your problem

Sample a hundred real queries and count tokens. If your median query is a handful of words, rewriting will not help much and you should be spending the effort elsewhere. If your median is a sentence with a question mark — anything with a chat interface in front of it — the rewrite stage is likely the cheapest available improvement, and the reason is structural: you are retrieving with a string that was written to be read by a person.