You added a reranker because retrieval quality was not good enough. The relevant passage kept landing at rank eight when it should have been first, and a reranker is the standard fix: pull a batch of candidates from your vector search, run a more careful model over them, and reorder so the best rise to the top. You wired it in, and the ordering did get sharper. The good chunks moved up. And yet the queries that were failing before are still failing. Not all of them, but the ones that mattered.
This is the quiet disappointment of reranking. It does exactly what it promises and still leaves you short, because it works on a shortlist it did not choose. A reranker can only reorder the candidates the first stage already handed it. If the passage that answers the question never made that shortlist, no amount of careful reordering will conjure it back. The reranker was never the stage that decided what you could find.
Two stages, and why there have to be two
Add a reranker and retrieval becomes two stages, and the split is not an accident. Plenty of RAG systems run a single stage, just vector search, or a hybrid of dense and keyword search, with no reranker at all. But the moment you add one, you have two stages, and they exist as two because the jobs, finding candidates cheaply and judging them precisely, have incompatible cost profiles.
The first stage is a bi-encoder. That is the ordinary vector search you already run: every chunk is embedded into a vector once, ahead of time, and at query time you embed the question and find the nearest chunk vectors. (A note on vocabulary, because the field is loose about it: the retrievable unit here is a chunk, a passage cut from a larger source. The retrieval literature confusingly calls each one a "document," and the vendor quotes below do too. I will say chunk.) The word "bi" is the point. The query and the chunk are encoded separately, never seeing each other, which is precisely what lets you precompute the whole corpus offline. It is cheap and it scales, and the price you pay is that the chunk was compressed into a single vector with no knowledge of the question that would later be asked of it.
The second stage is a cross-encoder, which is what most rerankers are. It takes the query and one candidate chunk together, in the same forward pass, and lets every word of the question attend to every word of the chunk before producing a single relevance score. That joint view is why it is more accurate. It is also why it cannot scale. There is nothing to precompute, because the score depends on the pairing, so you pay a full model inference for every query-and-chunk pair at the moment the query arrives. Cost grows linearly with the number of candidates. Score a thousand candidates and you run the model a thousand times.
The size of that gap is easy to underestimate. Pinecone puts a number on it: over forty million records, reranking every one with a small cross-encoder on a V100 GPU would take more than fifty hours to answer a single query, work that ordinary vector search does in under a hundred milliseconds. That is the whole reason the pipeline has two stages. You cannot run the accurate model over everything, so you use the cheap model to narrow millions down to a manageable candidate set, and spend the expensive model only on that set.
Which fixes the shortlist as the object the reranker acts on. The retriever decides who is in the room. The reranker only decides who wins the room.
What reranking moves, and what it cannot
Two words from information retrieval make the limit exact, and they are worth stating plainly because the whole argument rests on them. Precision is the fraction of what you retrieved that is actually relevant. Recall is the fraction of all the relevant material that you managed to retrieve. They are different questions. Precision asks "is what I got any good," recall asks "did I get everything I needed."
A reranker reorders a set. It can push the relevant chunks in that set to the top and shove the irrelevant ones down, which raises precision at the positions the model will actually read. It can even rescue recall you would otherwise lose one step later: if a relevant chunk was retrieved but ranked too low to survive the top-n cut, reranking can lift it above the line and into the model's context, where a weaker ordering would have dropped it. What a reranker cannot do is add a chunk that is not in the set at all. The relevant material the first stage never retrieved is gone before the reranker runs, and the reranker has no mechanism to notice it is missing, let alone recover it. That boundary is the ceiling: reranking can reorder and rescue anything inside the retrieved set, but it cannot lift the set's own edge.
Picture hiring. Your recruiter screens a thousand applications and forwards a hundred. An interview panel ranks those hundred and you make three offers. A strong reranker is a sharper panel: better at telling the excellent candidate from the merely good one, so a strong candidate the recruiter did forward, but a weaker panel would have overlooked, can still make your final three. What the panel cannot do is interview the brilliant applicant whose resume the recruiter cut on the first pass. That person is not in the room. The panel never sees them, cannot score them, cannot hire them. A more discerning panel does nothing about someone screened out upstream. That person is your missing recall, and the screening step, not the interview, is where they were lost.
This is why the reflex to answer bad retrieval by bolting on a stronger reranker so often disappoints. It is a fair instinct, and I have reached for it too, because the reranker is the most visible knob and turning it does produce a real improvement. But it aims one layer too high. If your misses are recall misses, the answer simply was not in the candidate set, then the stage you need to fix is the retriever: the embedding model, the hybrid dense-plus-keyword search that widens what the first stage can find, or the chunking that decided what was retrievable in the first place. A reranker cannot fix a passage that chunking fragmented into uselessness or that the embedding never placed near the query. The recall ceiling is set upstream, and the reranker lives below it.
The recall you throw away
Now the sharper part. It is not only that the reranker cannot lift the ceiling. The stages around it actively spend the recall you did retrieve.
Start with how the funnel is built. Pinecone describes the design goal exactly: "maximize retrieval recall by retrieving plenty of documents, and then maximize LLM recall by minimizing the number of documents that make it to the LLM." Read that twice, because it contains a deliberate discard. You widen the first stage on purpose, pulling far more candidates than you will use, precisely so the answer is somewhere in the pile. Then you spend the rest of the pipeline throwing most of that pile away. The recall you paid to gather is not handed to the model. It is narrowed, on purpose, at several points.
The reranker itself is the first narrowing, and it has a hard limit on what it will even look at. Managed rerankers impose a finite input window, often around a hundred or two candidates per call, and reject anything larger rather than quietly accept it. So a production system has to cap, batch, or drop candidates before they are scored, which means the width of your first stage is bounded not only by latency but by the reranker's own intake limit.
Then comes selection. The reranker returns an ordered list, but you do not pass all of it to the model. You pass the top few, because the language model has a finite context window and every token costs latency and money. A typical pipeline retrieves tens to low hundreds of candidates and hands the model a single-digit number. Anthropic's contextual retrieval setup, for one concrete example, retrieves the top one hundred fifty chunks, reranks them, and keeps the top twenty. Everything between rank twenty-one and rank one hundred fifty was retrieved, embedded, and scored, and then discarded before generation. That is recall you fetched and paid for and did not use.
Even the survivors are not used evenly. Language models do not read a long context with uniform attention. The "lost in the middle" work by Liu and colleagues found a U-shaped curve: a model uses information best when it sits at the very start or the very end of the context, and measurably worse when the same information sits in the middle, and this holds even for models built for long contexts. So the order the reranker produces matters twice. It decides which chunks survive, and it decides where they land in the prompt. Put your single best passage in the middle of twenty others and the model may underweight the very thing you reranked to the top. Reranking is not just a filter. It is placement, and placement is used unevenly by the thing you are placing for.
There is even a stage that discards recall as its explicit purpose. Diversity reranking, the classic form is Maximal Marginal Relevance from Carbonell and Goldstein, deliberately drops candidates that are near-duplicates of ones already selected, so the limited slots carry distinct evidence instead of five restatements of the same sentence. This is the most defensible discard, because a duplicate carries no new information, but it is worth seeing it for what it is: another place in the funnel where retrieved, relevant material is thrown out by design.
Add these up and the shape is clear. You widen the net to raise recall, then a cap, a context budget, an ordering effect, and a dedup pass each take a cut. The recall that reaches the model is a fraction of the recall you retrieved, and reranking sits in the middle of that funnel, doing precision work, unable to add back anything the narrowing removed.
When a reranker earns its cost, and when it does not
None of this means skip reranking. A cross-encoder over a good candidate set genuinely improves the ordering the model reads, and better ordering is not cosmetic. Anthropic reported that adding reranking on top of their contextual retrieval took the failed-retrieval rate on the top twenty chunks from 2.9% to 1.9%, part of a stack that started at 5.7% with plain embeddings. Those are their own evaluation numbers, so read them as directional, but the direction is real and the mechanism is sound: the model spends its attention on better-ordered, less-cluttered context.
The cost is equally real, and it is worth being honest about where it does not pay off. The BEIR benchmark, which tests retrieval models across many domains they were not trained on, found that cross-encoder rerankers give the best zero-shot generalization of the approaches studied, but at high computational cost, and that a plain keyword baseline remains surprisingly robust. Two things follow. First, the accuracy comes with a latency bill that scales with how many candidates you rerank, which is exactly why you cannot escape the recall ceiling by simply reranking more. Second, a reranker trained narrowly on one distribution can underperform on queries far from that distribution, so the gain is not guaranteed on your data. Measure it on your data before you assume it.
The latency bill is not abstract. The workhorse open-source cross-encoders make the tradeoff visible on their own model cards: moving from the six-layer MS MARCO MiniLM to the twelve-layer version nearly halves throughput, from roughly eighteen hundred candidates per second to around nine hundred sixty on the same hardware, in exchange for a difference in ranking quality too small to matter. On that benchmark, the larger model bought mostly latency, not better answers.
Where the reranker even lives is a choice too, and the platforms differ more than the marketing suggests. Weaviate, Elasticsearch, and OpenSearch can run reranking natively as a second stage inside the engine. Qdrant, notably, does not run a cross-encoder itself: it fuses results and supports late-interaction scoring, but the cross-encoder reranking happens in your application layer. pgvector has no reranker at all; it is a similarity index, and any reranking is something you build on top. If your mental model is "the database reranks," check, because for some of the stack it does not, and the stage has to live in your code.
There is also a middle option between the two extremes, worth knowing even if you do not reach for it. Late-interaction models like ColBERT encode query and chunk separately, so chunk representations can still be precomputed like a bi-encoder, but they keep a per-token representation and do a cheaper fine-grained match at query time, recovering some of the cross-encoder's precision without the full per-pair cost. The price is index size, since per-token vectors are much larger than one vector per chunk, though later versions cut that footprint substantially. It is not free either. It just moves the cost to storage instead of query-time compute.
Budget the whole funnel, and measure each stage
The fix is not a better reranker. It is to stop treating retrieve, rerank, select, and order as four independent knobs and start treating them as one budget with one job: get the answer in front of the model, in a position the model will actually use.
That reframes every setting. How wide you retrieve is a recall decision, and it is the one that sets your ceiling, so it deserves the most attention, not the least. How many candidates you rerank trades latency for the chance that a lower-ranked-but-relevant candidate gets promoted. How many you keep trades context cost against the risk of dropping the answer. How you order what you keep decides whether the model actually reads your best evidence or loses it in the middle. These are not separate tuning exercises. Widen the first stage and you may need to rerank and keep more to preserve the gain, or the extra recall you just bought gets discarded two stages later, unused.
And because the failure is invisible from the output, the discipline is to measure recall stage by stage rather than trust the final answer. The retrieval community already measures first-stage quality directly as recall at k, the fraction of the true relevant chunks that appear in your top k, traded against throughput. Evaluation tools for RAG, like Ragas, split the measurement in exactly the way this whole problem demands: context recall asks whether the retrieved set contained everything needed to answer, and context precision asks whether the relevant pieces were ranked near the top. Those are the two numbers the two stages own. If context recall is low, no reranker will save you, and you are wasting money reranking a set that does not contain the answer. If context recall is high but context precision is low, that is exactly the gap a reranker is built to close. Measured separately, the numbers tell you which stage is actually failing. Measured only at the output, they tell you nothing, because a fluent answer built on a missed chunk looks identical to a fluent answer built on the right one. It is the same silent shape the rest of the vector-search stack shares: the system returns something well-formed and confident, and only a direct measurement shows it returned the wrong thing.
The pattern underneath
Reranking is late-stage precision applied to an early-stage recall decision. That is the whole thing in one line. The reranker is a genuinely useful tool doing genuinely useful work, and it is placed after the moment that decided what was possible. It sharpens the order of what survived. It cannot reach back across the boundary to what was screened out, and the stages after it keep narrowing what survived down further.
So the leverage is upstream, in the unglamorous first stage. Raise the ceiling before you polish what is under it: a first stage that retrieves the answer more often, through better embeddings, hybrid search, and chunking that keeps answers intact, gives the reranker something worth reordering. Then the reranker does its real job, which is precision on a set that already contains the truth, not a rescue of one that does not. Get the order right, and mind where the good passage lands, because the model reads the edges better than the middle. Reranking pays off exactly when it is the last refinement on a strong retrieval stage, and disappoints exactly when it is asked to be the retrieval stage.
The reranker will hand you a confident, well-ordered list either way. Whether the answer is anywhere in it was decided one stage earlier.
A reranker cannot rank what retrieval never fetched. It reorders and rescues what the first stage found. It cannot lift the ceiling the first stage set.
Sources
The figures here are reported by the vendors and papers cited, each on their own evaluation, not independently verified. Read them as directional.
- Nogueira & Cho, "Passage Re-ranking with BERT" (arXiv 1901.04085).
- Khattab & Zaharia, "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT," SIGIR 2020 (arXiv 2004.12832); Santhanam et al., "ColBERTv2," NAACL 2022 (arXiv 2112.01488).
- Liu et al., "Lost in the Middle: How Language Models Use Long Contexts," TACL 2024 (arXiv 2307.03172).
- Thakur et al., "BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models," NeurIPS 2021 (arXiv 2104.08663).
- Carbonell & Goldstein, "The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries," SIGIR 1998.
- Manning, Raghavan & Schütze, Introduction to Information Retrieval, Cambridge University Press 2008 (precision and recall definitions).
- Anthropic, "Introducing Contextual Retrieval" (September 2024).
- Pinecone, "Rerankers and Two-Stage Retrieval"; Sentence-Transformers "Retrieve & Re-Rank" and the MS MARCO cross-encoder model cards; Cohere, Weaviate, Qdrant, Elasticsearch, OpenSearch, and pgvector documentation; Ragas context precision and recall metrics; ANN-Benchmarks (Aumüller et al., 2020).