Filtering before or after you retrieve

A legal team searches a contract repository for “termination for convenience”, restricted to agreements signed after a particular reorganisation. The unrestricted query returns the right clauses. Add the date restriction and the result set is empty — not “fewer results”, empty.

The clauses matching that date exist. The filter didn’t remove them; it removed everything the retriever had already decided to return, which is a different operation with the same syntax.

Query: "termination for convenience"      (no filter)

LEXICAL (BM25)                            DENSE (vector)
1. MSA 2019 — termination clauses     ✗   1. MSA 2019 — termination clauses  ✗
2. Vendor SOW 2020 §12                ✗   2. Convenience vs cause explainer  ~
3. Convenience vs cause explainer     ~   3. Vendor SOW 2020 §12             ✗
4. Reseller agreement 2018            ✗   4. Notice periods overview         ~
5. Termination notice template        ~   5. Reseller agreement 2018         ✗

Every top result predates the cut-off. Apply the predicate to this list and nothing survives. The post-2022 agreements were at ranks 60 and upward, because the corpus is dominated by older documents and relevance had no reason to prefer new ones.

Two places the predicate can go

Post-filter. Retrieve, then discard results failing the predicate. Simple, works against any retriever, and the returned count is unpredictable: you ask for ten and receive anywhere from ten to zero depending on how the surviving documents happened to rank.

Pre-filter. Restrict the search to the matching subset, then rank within it. You reliably get ten results, all valid, ranked by relevance among documents that satisfy the constraint — which is almost always what the user meant.

post-filter:  search(corpus, q)[:k]  →  keep(p)      # count unpredictable
pre-filter:   search(subset(corpus, p), q)[:k]       # count reliable

The distinction is not a performance detail. Post-filtering answers “of the most relevant documents, which satisfy the constraint”; pre-filtering answers “of the documents satisfying the constraint, which are most relevant”. Users ask the second question and systems commonly implement the first.

Why pre-filtering is harder than it looks

For a lexical index, a predicate is a familiar thing — term-based retrieval already works by intersecting posting lists, and a metadata term is another list to intersect. The engine was built for this.

For a dense index, the search is a similarity traversal over a structure built without reference to your metadata. Restricting it to a subset means either checking the predicate during traversal or searching a separate structure per subset, and both have consequences that belong to whoever operates the index. What matters at the query-strategy level is the observable behaviour: on a dense index, a highly selective pre-filter can degrade result quality even when it returns a full page of results. The traversal spends its effort in a region where few documents qualify, and the neighbours it does surface are the qualifying ones it happened to reach, not the nearest qualifying ones overall.

So the honest position is that dense pre-filtering is approximate in a way dense search alone already is, and the approximation gets worse as the filter gets narrower. Whether that matters is measurable on your corpus and not predictable from the outside.

The selectivity question

Selectivity — the fraction of the corpus a predicate admits — determines which strategy is even viable.

Broad predicates (a language, a document type covering half the corpus) are safe post-filters. Most retrieved results pass, so the count loss is small and the ranking is undisturbed.

Narrow predicates (one tenant among thousands, one week of a decade-long archive) cannot be post-filtered at any depth you can afford. If one document in ten thousand qualifies, no realistic retrieval depth reliably contains one.

Extremely narrow predicates are not a retrieval problem at all. A single document ID, one contract number, one customer’s records — fetch by key. Ranking a set of three is theatre.

The awkward middle is where the work is, and the practical move is to make the decision at query time rather than picking one strategy for all queries. If you can estimate the predicate’s cardinality cheaply — a count on the metadata store, or a maintained histogram — you can pre-filter narrow queries and post-filter broad ones and stop reasoning about it.

Filters have to be symmetric

If you run two retrievers, the predicate must be applied identically to both, and it is startlingly easy for it not to be. The lexical index has a native filter clause. The vector store has a different filter syntax, or supports fewer operators, or handles null differently. Someone implements both, they diverge on an edge case, and the merged list contains results that violate the constraint — sourced entirely from the retriever whose filter was weaker.

The fusion step will not notice. It has no view of the predicate; it merges what it was given.

Two habits that catch this: assert the predicate over the final list in your test suite rather than trusting each retriever’s clause, and log which retriever produced each surviving result so an asymmetry shows up as a source imbalance rather than as a mysterious relevance complaint.

Nulls, and the filter that removes the answer

The most common filter bug has nothing to do with pre versus post. A predicate on a field that is missing from some records excludes those records silently, because a comparison against null is neither true nor false in most query languages — it simply doesn’t match.

A document whose date failed to parse at ingest has no date. A restriction to documents after a given date drops it. So does a restriction to documents before that date. The document is unreachable by any date-bounded query, and the symptom is a retrieval failure with no visible cause — the document is in the index, both retrievers would rank it, and it never appears.

Check for it directly: count records where each filterable field is null, per field, and treat a non-trivial count as a bug in extraction rather than a fact about the corpus. When diagnosing a specific miss, re-running with filters off is the fastest single test available, and it isolates this class immediately.

Filters as a substitute for retrieval

Some constraints belong in the predicate and nowhere else, and pushing them into the query text is a category error that no retriever recovers from.

Numeric comparisons — “over fifty seats”, “under a specified threshold” — are not representable as similarity, and lexical matching on the numerals finds documents that mention the number rather than documents satisfying the comparison. Ranges, statuses, ownership, permissions, and versions are the same. A retriever ranks by textual relatedness; anything you can decide with a comparison operator should be decided with one.

Which means the query-transform step and the filter step are the same design decision seen from two sides: the useful work is pulling structured constraints out of the natural-language query and turning them into predicates, leaving the retriever the part it is actually good at.