Back to Search Quality

When Is a Reranker Worth It?

Dylan Couzon

·

August 23, 2026

Before you tune a reranker, use the pre-tuning checks to verify index state and set a labeled baseline.

Your candidate list can already contain documents your ranking never shows. Score those candidates as if they were perfectly ordered, then compare that with the score your pipeline returns today. The gap between the two is everything a better ranking stage could recover, so measure it before you reach for a model. Use nDCG@10, which grades the top 10 results and gives more credit to relevant documents near the top.

A wide gap means a better order is worth chasing. The deepest count measured here was 200 candidates. At that count, the nDCG@10 gap ran from 0.247 to 0.487 across the five datasets, and candidate depth shows how to measure it on your own collection.

Reranking covers several model families. This article measures cross-encoders, which read the query and candidate together as a single sequence. A classification head returns one relevance score for the pair. Joint reading lets the model capture token interactions that separately encoded query and document vectors miss.

Every candidate takes a forward pass at query time, which rules a cross-encoder out as a first stage and keeps it in the reranking slot. Late interaction models rerank from stored vectors instead, and the last section covers where they fit.

Test a Reranker in Three Steps

1. Establish the baseline the reranker has to beat: tuned fusion if you run hybrid search, your current ranking if you run dense-only or sparse-only. Confirm the documents your labels mark relevant reach the candidate list. A reranker only reorders what it receives; missing documents are a candidate depth or retrieval problem.
2. Rerank 10 candidates with the model you would actually serve, since model choice moved our results more than any other setting. Read its model card first for languages, domains, and context window. Reranking with FastEmbed shows the cross-encoder workflow and the available models. Compare the result with the first-stage baseline on held-out labeled queries.
3. Raise the candidate count only if the reranker wins. Measure throughput on your document lengths before making it part of the serving path.

Step 2 starts from the request your service already sends. Request the payload fields the reranker reads and your labels key on, and keep the fusion settings you serve today.

from qdrant_client import QdrantClient, models

# Both prefetches use the models the collection was indexed with.
from your_embedding_setup import dense_query, sparse_query

client = QdrantClient(
    url="https://YOUR-CLUSTER.cloud.qdrant.io",
    api_key="<your-api-key>",
)
query_text = "the query text"

fused = client.query_points(
    collection_name="products",
    prefetch=[
        models.Prefetch(query=dense_query, using="dense", limit=200),
        models.Prefetch(query=sparse_query, using="bm25", limit=200),
    ],
    # Your tuned fusion settings; k=2 with equal weights is the default.
    # RrfQuery needs Qdrant v1.17 or later and a client release that exposes it.
    query=models.RrfQuery(rrf=models.Rrf(k=2, weights=[1.0, 1.0])),
    limit=10,
    with_payload=["text", "doc_id"],
).points

Then score the same 10 candidates with a cross-encoder and sort them by that score. Any FastEmbed cross-encoder works here, so name the model you plan to serve. Xenova/ms-marco-MiniLM-L-6-v2 appears in the examples because it is the quickest to download.

from fastembed.rerank.cross_encoder import TextCrossEncoder

encoder = TextCrossEncoder(model_name="Xenova/ms-marco-MiniLM-L-6-v2")
scores = list(encoder.rerank(query_text, [point.payload["text"] for point in fused]))

# Sort by position so tied scores never compare the points themselves.
order = sorted(range(len(fused)), key=lambda i: scores[i], reverse=True)
reranked = [fused[i] for i in order]

You now have two orderings of the same 10 candidates. Score both with the nDCG@10 function from the pre-tuning article.

# relevance holds this query's labels, keyed by doc_id.
before = ndcg_at_k([point.payload["doc_id"] for point in fused], relevance)
after = ndcg_at_k([point.payload["doc_id"] for point in reranked], relevance)

Run that over your labeled queries, average the per-query difference, then check the interval around it before you trust the direction.

Compare with the Best First Stage

Compare the reranker against the strongest first stage you can build. Qdrant’s default reciprocal rank fusion (RRF) is already a solid baseline, and fusion tuned on your own labels is stronger. A reranker measured against the default can look like a win that tuning would have delivered for far less work at query time.

So tune fusion first, then make that tuned ranking the number the reranker has to beat on held-out labeled queries.

Each row in the following table reports the best of four cross-encoders on that dataset. The deltas show the nDCG@10 change over default RRF and over fusion tuned on the same candidates. MiniLM-L-6, MiniLM-L-12, and bge-reranker-base truncate each pair at 512 tokens. jina-reranker-v2 reads up to 1024 and was trained on a broader mix, including code.

The held-out column is the one that decides. It holds the share of 200 split-half draws where the gain survived on queries it was not selected on.

Ship a reranker gain only when it survives held-out validation. The two confirmed wins here held in 100% of the split-half draws, while the three unconfirmed results held in under half of them. When a positive result fails the split, add queries or keep fusion, and use the label-count table in the pre-tuning article to size that confirmation.

Keep fusion when the reranker loses to the tuned first stage. WANDS gained +0.039 over default RRF and still lost to fusion tuned on the same candidates.

Both confirmed wins came from the model whose window and training data fit the corpus, scoring the same candidates the other three saw. Even those wins closed part of the gap, recovering 46% of it on CodeSearchNet and 24% on DBPedia-entity.

On DBPedia-entity, fusion left the page for “(Just Like) Starting Over” at rank 49 of 200, even though every term of the query “John Lennon Yoko Ono album Starting Over” sits in the page’s first two sentences. Fusion reads ranks, and neither prefetch ranked the page high: 35th dense, 45th sparse, behind pages whose titles name both Lennon and Ono. The cross-encoder read the query and page as one sequence and put it first.

Diagnose a Loss Before You Stop

If the reranker loses at 10 candidates, go back to fit and measure it. Tokenize a sample of your query-document pairs with the model’s tokenizer. Compare the 95th percentile length with the model’s window: a longer pair gets truncated, and the model scores a document it only partly read. Then reread the card’s languages and domains against your corpus.

Two mismatches explain every loss in the table. Long queries are the first. ArguAna queries average 168 words, which leaves little room for the document inside a 512-token truncation. The 1024-token model turned that loss into a win, though it held in only 2.5% of the held-out splits, so the label set cannot confirm it.

A training-domain gap is the second. None of the three older models was trained on code, and jina-reranker-v2 flipped CodeSearchNet from a loss into the largest confirmed win in the table, scoring the same candidates the others saw.

If you find a mismatch, swap in a model whose window and training data fit your documents and rerun the 10-candidate test. If the model fits and still loses, keep the tuned first stage and spend the tuning effort elsewhere: on WANDS, tuned fusion beat all four models at every candidate count.

Set Candidate Count After a Win

Start with 10 candidates, and confirm on your labeled queries that the reranker beats tuned fusion before you change the count. Every configuration that trailed tuned fusion at 10 candidates still trailed it at 200, so a deeper list does not rescue a reranker that loses at 10. nDCG@10 grades the same top 10 results at every count, so the count changes only what the reranker gets to choose from.

The best nDCG@10 change over tuned fusion among the four models, by candidate count. A line above zero is a reranker win; WANDS never crosses it.

Step 1 confirmed that your relevant documents reach the candidate list. Run that same check at each count you are considering, before you run the reranker at any of them. The share of queries whose relevant documents are already in the candidate list limits how much increasing the count can help. Beyond that point, extra candidates only add documents that can push the relevant ones out of the top 10.

Rerank only at candidate counts where that share is still climbing. When the first stage finds relevant documents early, the share flattens quickly. In ArguAna, each query has one relevant document, and 90% of queries already included it within the first 25 candidates. Increasing the count to 200 raised that share to 98%, but turned the gain into a loss.

Where the share keeps rising, deeper reranking keeps finding documents the first stage buried. CodeSearchNet held the relevant document for 80.5% of queries at 25 candidates and for 90.5% at 200, and its gain kept climbing through 200.

The relevance gain can flatten before that share does, so take the smallest count that captures most of it. DBPedia-entity reached 96% of its eventual gain by 50 candidates, so going to 200 quadrupled the reranking work for the last 4%.

The shape you get depends on how many relevant documents your queries have and how well your first stage already ranks them. Measure it on your own labels rather than borrowing a count from these datasets.

Size Reranking for Production

Once relevance has settled the candidate count and the model, measure query-candidate pairs per second and tail latency on the hardware you plan to deploy, using representative document lengths and concurrency.

The table shows CPU throughput for the four FastEmbed cross-encoders, listed by their full model IDs and measured in one process on an Apple M5 Pro with 15 threads. The last column converts that rate to whole queries at 100 candidates each.

Document length explains each range. DBPedia-entity has short entity abstracts, while SciFact has full paper abstracts.

Weigh those rates against the held-out gain. At 100 candidates, one CPU process spends between half a second and five seconds per query with the three smaller models, where the second prefetch behind tuned fusion added 0.6 to 1.5 ms in the same setup. The 10-candidate test itself stays fast even on CPU, at 47 to 156 ms per query with the smallest model, so run it before you plan any serving work.

Pick the model on fit rather than size. bge-reranker-base and jina-reranker-v2 are nearly the same size, and only the second ever beat tuned fusion. Training data and context window separated them.

Some models only reach a usable rate on a GPU. The jina-reranker-v2 ONNX export runs one CPU thread at a time through an attention kernel, which is why its row reads under 2. On a GPU through PyTorch it ran at 32 to 310 documents per second, and the quality numbers here come from that run. It ships under a CC-BY-NC-4.0 license, so check the terms first.

Use Other Stages for Different Problems

Match the stage to the symptom you see in your results.

Maximal marginal relevance trades relevance for diversity, and nDCG does not reward the diversity it adds, so measure the direction on your own labels before shipping it.

Grouping fits collections that store each chunk of a document as its own point. query_points_groups with group_by on the document ID field returns the best chunk per document, so one long document cannot fill the first page. The grouped field needs a payload index; without one on document_id, Qdrant Cloud returns a 400.

Formula Query rescores the same candidates with an expression over payload fields, such as recency or popularity, and needs a payload index on each field the formula references.

Test a late interaction model when a cross-encoder is too slow. Document vectors are built at ingest, so only the query goes through the model per request, and the rescoring runs inside Qdrant in the same multi-stage query. Storage grows to a vector per token. Multivectors and Late Interaction walks the full setup.

What to Tune Next

After a win, the work moves to throughput, where the candidate count you can serve decides how much of the gain survives. After a loss, the gap is still there and the candidates are what to change, so revisit retrieval and candidate depth before adding another ranking stage.

Next, if memory is the constraint, measure what memory placement and rescoring add to query latency.