When a reranker earns its latency

Fusion gave you a merged list. The right document is in it, at rank seven, and you’re passing the top three to the model. So retrieval worked and the answer is still wrong.

This is the specific failure a reranker addresses: the candidate set is good and the ordering isn’t. If that’s not your failure, a reranker won’t help and will cost you latency anyway.

Why it ranks better

Retrieval, in both signals, scores the query and the document separately and then compares the results.

A dense retriever embeds the document at index time — long before your query exists — and embeds the query at search time, then measures distance between two vectors computed in isolation. A lexical retriever scores term statistics that were also computed in advance. Neither ever looks at the query and the document together.

That independence is what makes retrieval fast. Document representations are precomputed and indexed, so search touches an index rather than the corpus. It’s also what limits quality: the document’s representation had to be useful for every possible query, so it’s necessarily generic.

A cross-encoder reranker does the opposite. It takes the query and one passage as a single input and produces a relevance score for that specific pair. Every token of the query can attend to every token of the passage.

retrieval:   f(query) · g(passage)          → precomputable, fast, generic
reranking:   h(query, passage)              → per pair, slow, specific

Which is why it catches things retrieval can’t:

Query: "does the free plan include SSO?"

AFTER FUSION                                    AFTER RERANK
1. Single sign-on setup guide          ~        1. Plan comparison table        ✓
2. Enterprise SSO configuration        ✗        2. Free plan limitations        ✓
3. Authentication overview             ~        3. Single sign-on setup guide   ~
4. Free plan limitations               ✓        4. Authentication overview      ~
5. Plan comparison table               ✓        5. Enterprise SSO configuration ✗

Both retrievers latched onto “SSO” and returned SSO documentation. The question was about plan entitlement, and the documents that answer it are about plans. A cross-encoder reading the query alongside each passage can tell that a plan comparison table answers “does the free plan include X” better than a configuration guide for X does.

The general pattern: reranking is strongest on multi-constraint queries, where relevance depends on the interaction between parts of the query rather than on overall topical similarity.

What it costs

The independence you gave up is the cost.

It’s a model pass per candidate. Rerank fifty passages, run the model fifty times. This does not benefit from an index; it is linear in the number of candidates, and each pass is roughly the cost of processing the query plus a passage of text.

It’s usually the dominant latency in the pipeline. Retrieval against a built index is fast; a cross-encoder pass over a few dozen candidates is not, and it happens after retrieval rather than in parallel with it. If you have a latency budget, this is where most of it goes.

It’s an extra dependency in the request path. Self-hosted, it needs a GPU or it will be slower still. Hosted, it’s a network call that can fail, and you need a defined behaviour when it does — usually falling back to the fused order.

Cost scales with candidate count and passage length. Both are knobs, and both trade quality for speed.

Where it belongs

Standard shape:

query
  ├─ lexical retrieval  → top 50
  └─ dense retrieval    → top 50
              ↓
       fuse (RRF) → top 50 candidates
              ↓
       cross-encoder rerank → score all 50
              ↓
       take top 5 → the model

Two design consequences worth stating explicitly.

Retrieval’s job changes to recall. Once a reranker exists, retrieval only has to get the right document somewhere in the candidate set. Precise ordering is the reranker’s problem. So retrieve more aggressively than you would otherwise — a wider net that the reranker sorts out. This also makes fusion quality less critical, which is a real simplification.

Candidate count is your quality/latency dial. More candidates means more chance the right document is present and a proportionally slower rerank. This is the single parameter to tune, and the useful range is corpus-dependent: measure where recall stops improving and stop there, because everything past that point is latency you’re paying for nothing.

When to skip it

Your recall is the problem, not your ordering. If the right document isn’t in the candidate set, reranking cannot conjure it. Check this first — it’s the most common reason a reranker disappoints. Take failing queries, look at the top 50 candidates, and ask whether the answer was there at all. If it wasn’t, fix retrieval; a reranker is the wrong layer.

Your latency budget is tight. An autocomplete or a typeahead cannot afford this. An answer-generation pipeline where the model call dominates anyway usually can.

Your candidate lists are already good. Small corpus, distinctive documents, queries that map cleanly. If the top three after fusion are consistently right, there’s nothing to reorder.

You need scores you can threshold on. Cross-encoder scores are generally better calibrated than fused ranks for deciding “is anything here relevant at all” — but calibration is model-specific, and if you’re relying on a threshold you have to establish it empirically rather than assuming one.

Practical notes

Rerank the same text you’ll pass to the model. Scoring a truncated passage and then sending the full one means you ranked something other than what you’re using.

Watch passage length against the reranker’s input limit. Long passages get truncated, and the truncated part is scored as absent. A long document whose relevant paragraph is at the end can be scored on its irrelevant beginning.

Keep the pre-rerank order as a fallback. When the reranker is unavailable or times out, serve the fused list rather than failing the request.

Log both positions. Record where each final result sat before and after reranking. If the reranker is consistently promoting things from rank 40, retrieve more. If it never moves anything, it isn’t earning its latency and you should turn it off.

Cheaper alternatives exist and are worth trying first. Late-interaction models sit between bi-encoders and cross-encoders — more precise than plain vector search, cheaper than full cross-encoding, at the price of a larger index. Using a language model as a reranker via prompting is also possible and is generally slower and more expensive than a purpose-built cross-encoder, so treat it as a prototype rather than a design.

Measure the thing that matters. Not “did reranking change the order” — it always does — but whether the final answers improved. It’s an extra model in the request path, and it should have to justify itself on end-to-end quality. If your failures turn out to be recall failures rather than ordering failures, work out which signal missed instead.