Which field the match landed in

A property listing search is asked for “garage conversion”. One document is titled “Garage conversions: planning and building regulations”. Another mentions the phrase once, in a footnote about a neighbouring property. Lexical retrieval ranks the second one higher, because it is short and the term therefore accounts for more of it.

The retriever is behaving exactly as specified. It was given a single blob of text per document and asked where the term occurs most densely. Nobody told it that one occurrence was in a title.

Query: "garage conversion"

SINGLE-FIELD INDEX                        TITLE-WEIGHTED INDEX
1. Note: adjacent garage conversion   ✗   1. Garage conversions: planning     ✓
2. Outbuilding permitted development  ~   2. Converting a garage: costs       ✓
3. Garage conversions: planning       ✓   3. Outbuilding permitted dev.       ~
4. Parking provision guidance         ✗   4. Note: adjacent garage conv.      ✗
5. Converting a garage: costs         ✓   5. Parking provision guidance       ✗

What a field is, and what concatenation destroys

Documents have structure: a title, headings, a summary, body text, tags, author, product attributes. An index can either preserve that structure as separate fields or flatten it into one text value.

Flattening loses two things:

Where the match occurred. A term in a title is a claim about what the document is about. The same term in the body is a mention. Term-based scoring cannot distinguish them once they are in one blob, because the only positional information it retains is frequency.

Per-field length normalisation. Scoring functions discount matches in long text, because a term appearing once in a paragraph is more indicative than once in a chapter. Flattened, every match is normalised against the whole document’s length, so a title match — occurring in a field of five words — is normalised as though it occurred in a field of five thousand. The strongest available signal is diluted precisely by the presence of the body text.

So field structure is information the corpus already has, and flattening is the decision to discard it. That framing is useful because it reframes field weighting as recovering something rather than adding a heuristic.

Two ways to weight fields, which behave differently

Search each field separately and combine the scores. Straightforward, and it has a specific defect: a query whose terms are split across fields scores badly everywhere. “Garage conversion Bristol” with “garage conversion” in the title and “Bristol” in the body matches each field partially and neither fully, so per-field length normalisation penalises both. Searching fields independently is a per-field competition, and a query spanning fields loses it.

Score the fields as one weighted pseudo-document. Combine term statistics across fields with weights before scoring, so a term appearing in a weighted title counts as if it appeared more times, and length normalisation is applied to a weighted combination of field lengths. This handles cross-field queries correctly, and it is what the field-aware members of the scoring family do.

The distinction matters in practice because the first is what you get by naively running several queries and adding results, and the second is what you get from an engine’s field-aware scoring. They produce different rankings on exactly the queries users type.

Where dense retrieval stands

Dense retrieval has no fields. It has one vector per indexed unit, and that unit’s text either includes the title or it doesn’t.

Which yields a decision with no correct universal answer: prepend the title and headings to the text you embed, or don’t. Prepending gives the passage’s vector some of the document’s topical context, which usually helps when passages are small and self-referentially vague. It also dilutes the passage’s own content, and if every passage from one document shares a prepended title, all of that document’s vectors are pulled toward each other — which increases the chance that a handful of near-identical results crowd out everything else.

There is no field-weighting knob on the dense side to tune afterwards. It is fixed at indexing time and changing it means re-embedding. Because of that asymmetry, field-level control is largely a lexical capability, and it is one of the concrete reasons the lexical signal remains useful in a hybrid pipeline rather than merely being the identifier fallback.

How boosts go wrong

Field weights are the easiest thing in a search stack to over-tune, and the failures are recognisable.

Boosting the title until only titles matter. A large title weight turns your search into a title search. Documents whose titles are generic — “Overview”, “Introduction”, “FAQ” — become unreachable regardless of their content, and documents with keyword-stuffed titles dominate. This is the most common outcome of tuning a boost against a handful of example queries.

Boosting a field that is not always populated. If a summary field exists on some documents, weighting it advantages those documents systematically. That is a ranking bias determined by whoever filled in metadata, not by relevance, and it looks like a mysterious preference for older or newer content depending on when the field was introduced. The same null-handling problem that breaks filters, expressed as a ranking distortion instead of an exclusion.

Boosting tags or keywords fields. Author-supplied keywords are an incentive surface. If anyone in your organisation knows the field is boosted, it will be stuffed, and the ranking will reflect internal politics.

Tuning weights per query class without saying so. Weights derived from a sample of one kind of query are wrong for the others, exactly as fusion weights are. If your sample is navigational queries — people searching for a document they know exists — a heavy title boost will look excellent and will damage every informational query.

Fields as filters versus fields as signals

Worth separating, because they get conflated in the same configuration file.

Some fields are predicates: status, tenant, date, type. These belong in filters, decided by comparison, never boosted. Boosting a status field means low-relevance documents with the right status outrank high-relevance documents with the wrong one, which is not what anyone meant.

Some fields are relevance signals: title, headings, summary, product name. These belong in weighted scoring.

Some are both, and need to appear twice — an author field can be a filter for “documents by X” and a weak signal when a name appears in the query. Handling that properly means indexing the field in two forms, one analysed for matching and one kept exact for filtering, and it is a routine source of configuration bugs when only one form exists.

Telling whether it’s your problem

The diagnostic is to look at where matches land in your failing queries. For a sample of queries with a known correct document, record whether the correct document had a query term in its title, and whether the documents ranked above it did.

The pattern to look for: the correct document matched in the title and lost to documents that matched in the body. If that is common, you are flattening structure you have. If it is rare — if your corpus’s titles are uninformative, generated, or absent — field weighting has nothing to offer and you should be looking at depth or ordering instead.

Then check the reverse, because it is the failure your fix will introduce: documents with keyword-heavy titles and thin content ranking above substantive documents. If you already see that, your title weight is too high, and lowering it will do more than raising anything.