Two Signals

Making a term mandatory

An analyst searches a filings archive for “restatement of deferred revenue 2023”. The top results discuss deferred revenue at length, in the right accounting context, from the right kind of document — and none of them mentions a restatement.

The word that made the query a specific request contributed a fraction of each document’s score, and documents that matched the other three words heavily outscored the one document that matched all four. Nothing is broken. Term scoring is a sum, and a sum does not require every addend.

Query: "restatement of deferred revenue 2023"

LEXICAL (BM25)                            DENSE (vector)
1. Deferred revenue recognition       ~   1. Revenue recognition policy       ~
2. Deferred revenue, FY2023 notes     ~   2. Deferred revenue, FY2023 notes   ~
3. Revenue policy, deferred items     ✗   3. Prior period adjustments          ~
4. Restatement of prior periods, 2023 ✓   4. Deferred revenue recognition     ~
5. Deferred tax and revenue timing    ✗   5. Accounting change disclosures    ~

Why additive scoring buries the decisive term

A term’s contribution depends on how rare it is in the corpus and how often it appears in the document. Both work against you here.

“Restatement” appears once in the relevant filing — a heading and a paragraph. “Deferred revenue” appears throughout documents that are entirely about deferred revenue. So a document repeating three of your four terms many times accumulates more score than a document containing all four, one of them once. Rarity weighting raises the price of the decisive term; it does not make it compulsory.

The dense side has a different route to the same result. One embedding represents the whole query, and a single word within a five-word query shifts that vector modestly. The nearest documents are those matching the query’s dominant topic, and “restatement” is a qualifier, not the topic. Both signals treat your decisive word as one contribution among several.

Required terms as a hard constraint

Every lexical engine can express “the document must contain this term”, separately from scoring — the mechanism behind the +term convention and behind boolean clauses in structured query languages. The term becomes a filter, and scoring then orders only the documents that pass.

Applied above:

Query: +restatement deferred revenue 2023

1. Restatement of prior periods, 2023  ✓
2. Accounting change disclosures       ~
3. Restatement policy summary          ~
   (candidate pool: documents containing "restatement")

The ranking is unchanged in character; the pool is smaller and correct.

Two things this is not. It is not a relevance boost — boosting raises a term’s weight and still permits documents lacking it to win, which is why boosting is the wrong tool for this problem and is usually reached for first. And it is not phrase matching, which additionally constrains word order and adjacency — a different mechanism for a different need.

What it costs

Zero results become possible, and they become likely. The moment a term is mandatory, a corpus that words the concept differently returns nothing. If the filings said “revision” rather than “restatement”, the constrained query has an empty result set where the unconstrained one had a mediocre list. A required term converts a ranking failure into an empty result — sometimes an improvement, always a change in failure mode, and empty results need handling rather than tolerating.

It interacts with the analyser. “Must contain” means must contain after analysis. If your stemmer maps “restatement” and “restated” to one root, the requirement is looser than it reads; if a hyphenated or camelCase form tokenises unexpectedly, the requirement can be unsatisfiable and no document in the corpus passes.

It is brittle against vocabulary mismatch, which is the one problem lexical retrieval was already worst at. A required term is a bet that the corpus uses your word.

It can be satisfied trivially. A document containing the term once, in a footnote or a boilerplate disclaimer, passes the constraint and enters the pool. Requirement is presence, not aboutness.

The hybrid problem

Here is the part specific to running two retrievers: dense retrieval has no notion of a required term. There is no clause to add. A vector search returns nearest neighbours, and “must contain this string” is not expressible in that operation.

So a required term applies to one half of your pipeline, and the consequences show up in the merge:

  • The lexical list respects the constraint. The dense list does not.
  • Fused, the final list contains documents lacking the required term — supplied entirely by the dense retriever, which had no way to know.
  • Fusion cannot repair this, because it has no view of the constraint; it merges the lists it is given.

Three defensible responses:

Enforce the constraint after fusion. Filter the merged list for the required term. Simple, and it throws away dense results after paying for them — and if the dense retriever supplied most of the candidates, the surviving list can be thin. The same post-filtering count problem as any late predicate.

Move the term into metadata and pre-filter both retrievers. Only works when the requirement is structural rather than lexical — a document type, a status, a year — but when it does work it is the correct answer, because both retrievers can apply a metadata predicate and neither can apply a text requirement.

Route the query to lexical alone. If the user has expressed a hard requirement, they are asking for term matching, and that is a routing signal as clear as a quoted phrase. You lose the dense contribution and gain a result set that means what it says.

Who gets to require a term

Explicit user syntax. Quotation marks, a leading plus, an advanced-search field. The user has stated a requirement, and honouring it is straightforward — the only obligation is telling them when it produced nothing, and offering the unconstrained results as an alternative rather than silently relaxing it.

Inferred from the query, which is where it gets dangerous. A system that decides which of your words were important is guessing, and a wrong guess is a hard constraint on the wrong term — an empty or misdirected result set with no explanation. If you infer, infer narrowly: a token matching an identifier pattern, or a term the user repeated, and nothing more speculative than that.

A softer middle exists and is often better than either: require nothing, but heavily weight the rarest query term and re-run with the requirement only if the unconstrained top results all lack it. That turns a hard constraint into a conditional second pass, at the cost of one extra retrieval on the queries that need it.

Telling whether it’s your problem

Look for the specific signature: the correct document is in the corpus, contains every query term, and is outranked by documents missing one of them. That is not a depth problem and not a fusion problem, and it will not respond to any weighting change.

The measurement, on a sample of failing queries: for each, check whether the top-ranked results contain all query terms. Count how often the answer’s distinguishing term is absent from the results above it. If that count is high, term requirements are your intervention, and the queries where it happens will usually share a shape — a topic word plus a qualifier — that tells you exactly which term to require.

lexicalfusion