Using an LLM as a reranker

You have a candidate set that needs better ordering, no labelled data to train on, and no appetite for hosting another model. The available shortcut is to hand the candidates to a general language model and ask which ones answer the question.

It works, in the sense that the ordering improves on plausible queries. It also introduces four behaviours that a purpose-built cross-encoder does not have, and each of them will eventually appear in your incident log.

Query: "can a tenant sublet a single room without the landlord's consent"

FUSED CANDIDATES                          AFTER LLM RERANKING
1. Assignment and subletting clause   ~   1. Subletting part of a property    ✓
2. Notice requirements for tenants    ✗   2. Assignment and subletting clause ~
3. Subletting part of a property      ✓   3. Consent, when it may be refused  ~
4. Consent, when it may be refused    ~   4. Lodgers and licence agreements   ~
5. Lodgers and licence agreements     ~   5. Notice requirements for tenants  ✗

The improvement is real, and it comes from the same mechanism a cross-encoder uses: the model reads the query and the passage together and can attend to “single room”, “without”, and “consent” jointly rather than scoring them as separate contributions.

Three ways to ask, with different properties

Pointwise. Score each candidate independently — one call per candidate, or a small batch each. Simple, parallelisable, and comparable across candidates only to the extent the model’s scoring is stable, which is less than you would like. Its virtue is that adding a candidate doesn’t change any other candidate’s score.

Listwise. Put all candidates in one prompt and ask for an ordering. One call, so it is cheaper, and the model can compare candidates against each other, which is genuinely more information than pointwise scoring has. Its costs are the interesting ones: the candidate set must fit in the prompt, the output is a permutation that the model may render incorrectly, and the order you present candidates in affects the order it returns.

Pairwise. Ask which of two candidates is better, repeatedly. The most reliable per-judgement and the worst per-query cost, since the number of comparisons grows faster than the candidate count. Used mainly in tournament arrangements over a small set.

Most deployments use listwise for cost reasons and inherit its position bias, so that one deserves its own treatment.

Position bias, and how to detect it

Models attend unevenly across a long prompt. A candidate placed first or last in a list is treated differently from one in the middle, independent of its content. That means the retrieval ranking you feed in leaks into the reranked output, and if you feed candidates in fused order, the reranker will partly reproduce the ordering it was supposed to correct — while appearing to have judged it.

The detection is straightforward and worth running once before trusting the stage: rerank the same candidate set twice, with the candidates shuffled differently, and compare the outputs. A stable reranker returns the same ordering. If the two differ substantially, the position of a candidate is influencing its rank, and any improvement you measured is partly an artefact of your input order.

Mitigations, all partial: shuffle the input rather than presenting fused order, so the bias is at least not correlated with the ranking you’re trying to fix; rerank in overlapping windows and merge; or run twice with different orders and combine the two rankings, which is another use for rank-based fusion and roughly doubles the cost.

Non-determinism

The same query and the same candidates can produce different orderings on different runs. Even at the most deterministic settings a provider offers, identical output across runs is not something you can rely on, and a model version change will move behaviour more than any parameter.

Consequences worth planning for rather than discovering:

• A user retrying a query gets different sources with no explanation.
• Comparing two pipeline configurations requires distinguishing a real difference from run-to-run variance.
• A cached answer and a fresh answer to the same question can cite different passages.
• Regressions arrive with provider updates you did not schedule.

A purpose-built cross-encoder has none of this. It is a fixed function of its inputs, and that predictability is a large part of what you give up.

Cost, honestly

Per-query cost scales with total candidate text. Both pointwise and listwise arrangements put every candidate’s text through a model. That is a per-query cost proportional to how much you retrieved, on every query, forever — and candidate passages are not short.

Latency lands in the critical path. The reranking call precedes generation and cannot overlap with it. Listwise is one round trip; pointwise is many, parallelisable up to your rate limits.

Rate limits become a capacity constraint. Your search throughput is now bounded by an external service’s quota, and the failure mode under load is a queue in front of your search box.

The output must be parsed. A model asked for a permutation can return a malformed one, invent a candidate identifier, or omit candidates. Validate against the input set and fall back to the input order on any mismatch — which is one of those pieces of defensive code that is unglamorous and load- bearing.

Where the trade is worth it

Before you have labelled data. No training set, no fine-tuning, no model to host. For a system with no traffic yet, this is the fastest route to a working ordering stage, and it can be replaced later.

On small candidate sets with high value per query. A handful of candidates, a query whose answer matters, and a latency budget that tolerates a round trip. Legal, medical and technical support search often fit.

When you need a reason, not just a rank. A model can state why a passage does or doesn’t answer the question. As a debugging aid during development that is genuinely useful — it will tell you that the passage discusses subletting the whole property rather than a room, which is the distinction your retrievers missed. Treat those explanations as development aids rather than as ground truth; a fluent justification for a wrong ordering is exactly what this technology produces.

When the queries need reasoning that similarity cannot express. Conditions, exclusions, negations, multi-clause constraints. This is where the gap over both retrievers is largest.

Where it isn’t

High query volume with a tight latency budget. A dedicated cross-encoder is the cheaper answer per query and the more predictable one.

When your failures are recall failures. No reranker of any kind adds a document to the candidate set, and this one costs the most to learn that lesson with.

When determinism is a requirement. Regulated or audited environments where the same query must produce the same sources.

As a way to avoid tuning retrieval. A strong reranker on a weak candidate set is the most expensive possible way to be wrong, and it hides the underlying problem well enough that nobody looks at which signal failed.

Measure it against the cheaper option

The comparison that matters is not “with reranking versus without”. It is against a dedicated cross-encoder on the same candidate sets, because that is the alternative you are declining.

Same queries, same candidate sets, three orderings — fused, cross-encoder, model — and the rank of the known-correct passage under each. Then weigh the difference against the difference in cost and determinism, on your traffic. That comparison, unlike a general claim about which reranking approach is better, will actually decide the question for your system.