Reranking Models as a RAG Retrieval Layer

A retriever pulling back the right 50 chunks means nothing if the LLM only reads the top 5, and the correct answer sits at position 23. That's the failure mode this piece is about: not missing information, but buried information. Retrieval did its job. Ranking didn't, and most teams debugging a bad RAG answer look in the wrong place because of it.

The symptom is a ranking failure, not a retrieval failure. The correct document exists in the candidate pool but sits too far down the list for the LLM to see it, because most pipelines pass only the top 3 to 5 results into the prompt. Fixing it requires a second pass over the retrieved candidates using a cross-encoder reranker, which scores each query-document pair jointly instead of comparing two independently compressed vectors. A cross-encoder rescoring the top 50 to 100 candidates adds roughly 50 to 150 milliseconds and, per Pinecone's analysis, a three-stage BM25 plus dense plus reranking setup showed a 48% improvement in retrieval quality over any single method alone, with lower hallucination rates downstream as the metric that actually matters.

Picture a support bot that keeps citing a deprecated API version. Somewhere in its candidate pool, the current docs page exists. It just sits at position 23, buried under a pile of near-misses that share more surface-level keywords with the query. The LLM never sees it, because most pipelines only pass the top 3 to 5 results into the prompt. That's not a recall failure, it's a ranking failure, and it's structural: the fix isn't better retrieval, it's a second pass that actually reads the candidates instead of just fetching them.

Why does the LLM cite wrong information when the right document is already in the retrieval pool?

First-stage retrieval, whether it's BM25 or a vector search over embeddings, has one job: scan a huge corpus fast and pull back a reasonable candidate set. To do that fast, it has to compress. A bi-encoder turns each document into a single vector, computed ahead of time and stored, so a query at runtime just needs a cosine similarity check against a few million pre-baked vectors. That's why it runs in single-digit milliseconds even over large corpora.

Compression costs precision, and there's no getting around that trade. Squeezing a whole document into one vector throws away detail. The query and the document never actually meet: they get scored in isolation, then compared after the fact. That's fine for figuring out which 100 documents are roughly in the neighborhood. It's not fine for deciding which 5 of those 100 actually answer the question.

That's the job of stage two: a cross-encoder. Instead of encoding query and document separately, it concatenates them into one sequence and runs the whole thing through a transformer together. Every query token attends to every document token, so the model sees the interaction directly instead of comparing two vectors that happen to sit near each other in space. That produces a sharper relevance score, but it costs more, because nothing can be precomputed. The score only exists for that specific query-document pair.

The standard production shape looks like this: a bi-encoder retrieves the top 100 candidates from the full corpus very quickly, then a cross-encoder rescores those 100 in roughly 50 to 150 milliseconds. The cross-encoder stays affordable precisely because it never touches the full corpus, only the shortlist. Most pipelines end up narrowing 50 to 100 retrieved candidates down to the 3 to 5 that actually reach the LLM. Two stages, two different jobs: one built for recall, one built for precision.

Where late-interaction models fit between the two extremes

Between "one vector per document" and "score everything jointly" sits a middle option called late interaction, and ColBERT is the model that defined the category. Instead of collapsing a document into a single vector, ColBERT keeps one small embedding per token. At query time, it runs a MaxSim score, matching query tokens against document tokens directly, without a full joint forward pass through a transformer.

The latency gap here is the whole reason ColBERT gets used at all. Its p50 latency lands around 23 milliseconds. A standard cross-encoder running at 40 queries per second can see its p99.9 latency blow past 21 seconds. That gap only shows up at scale, but once it shows up, it decides whether an architecture is even usable in production.

The trade-off is storage. Keeping a vector per token instead of one per document inflates index size by one to two orders of magnitude. Later versions of ColBERT use residual compression to bring that down, but it's still heavier than single-vector storage. In exchange, ColBERT reported speedups of two orders of magnitude over earlier BERT-based rankers, while holding quality that stays competitive with heavier approaches.

Late interaction earns its keep in one specific spot: pipelines where cross-encoder latency would blow through the SLA, but a plain single-vector retriever isn't precise enough on its own. This isn't a question of which model scores higher on a benchmark. It's a budget question, how many queries per second, and how much storage the index can afford.

The production model landscape in 2025–2026

The reranker market has settled into a few clear lanes: hosted, self-hosted, cross-encoder, late-interaction, and generative. Hosted is the wrong default for anyone with real volume, but it's the right starting point, and most teams should start there before deciding they need to self-host.

Cohere Rerank 4 comes in Pro and Fast variants, covers over 100 languages, and gives predictable latency without any infrastructure to manage. Voyage's rerank-2.5 is the current flagship on the hosted side too: API-hosted, known for high accuracy, and Voyage offers 200 million free tokens to try it before committing.

For teams that outgrow hosted or need data to stay in-house, BGE Reranker v2-m3 from the BAAI BGE family is the strongest open-license pick at 568 million parameters and over 100 languages (the same family has bge-reranker-base at 278 million and bge-reranker-large at 560 million for lighter deployments). Mixedbread's mxbai-rerank-large-v2 runs on a Qwen-2.5 architecture at 1.5 billion parameters and posts some of the top BEIR scores among open-weight models; mxbai-rerank-base-v2 at 0.5 billion parameters covers lighter loads. ColBERTv2 stays the canonical late-interaction self-host option, with multilingual variants available through separate community models. Qwen3 Reranker ships in 0.6B, 4B, and 8B sizes and scores relevance by comparing the probability the model assigns to generating "yes" versus "no" for a given pair.

Then there's a category that plays by different rules entirely: LLM-based reranking. RankGPT-style approaches hand a set of passages to a generative model and ask it to produce a ranked list outright. RankZephyr distills that behavior into a smaller model, and RankLLM ties RankVicuna, RankZephyr, and RankGPT together under one framework. These can beat cross-encoders on accuracy, but they run slower and cost more per query, so they only make sense when result quality is worth the compute bill. And worth flagging plainly: zero-shot LLM rerankers lose ground fast to fine-tuned cross-encoders the moment the data drifts out-of-domain. That's not a minor caveat, it's the reason most production systems still run a cross-encoder underneath, not an LLM.

For a size reference on the lighter end, cross-encoder/ms-marco-MiniLM-L6-v2 scores 74.30 nDCG@10 on TREC DL19 and processes 1,800 documents per second, a useful baseline for what a small, fast model can do.

The real decision, before any model-picking, is hosted versus self-hosted, and bigger is no longer the safe assumption on either side. The 0.6B-parameter class of open-weight models has closed most of the gap with much larger ones, which means paying for a heavier model by default is usually just paying for nothing. As a rough map: license constraints point toward BGE v2-m3, Qwen3, or mxbai; multilingual needs point toward Cohere; latency-critical systems point toward ColBERT or MiniLM-class models; accuracy-first, cost-tolerant use cases point toward an LLM reranker.

What benchmarks show about the accuracy gains reranking actually delivers

The numbers back up the architecture, and they're not close. Fusing BM25 with dense embeddings already beats either method alone on hard retrieval sets, and layering a cross-encoder on top adds another 5 to 15 points of MRR on the harder evaluation sets.

Pinecone's analysis found a 48% improvement in retrieval quality from a three-stage BM25 plus dense plus reranking setup, compared to any single method alone. ZeroEntropy's zerank-1 reported a 28% gain in NDCG@10 over baseline retrievers, and that gain tracked with measurably lower hallucination rates downstream in RAG applications, which is really the number that matters.

The TREC 2024 RAG Track pushed this further with a full cascade: monoT5-3B reranked the top 3,000 documents from initial retrieval, fused with RRF, and then RankZephyr reranked the top 100 from that output and fused again. Layered reranking, stacked twice, on a competitive academic benchmark. And historically, cross-encoder rerankers were the approach that first took the top spot on the MS MARCO passage-ranking leaderboard, substantially outperforming BM25.

The gain isn't fixed, and treating it as a flat percentage across every use case is the mistake to avoid. It scales with how hard the retrieval problem is to begin with. A corpus full of clean keyword matches doesn't have much room to improve. A corpus full of near-duplicate, ambiguous, or cross-domain content has a lot of room, and that's exactly where reranking earns its cost. The metric that actually matters operationally isn't the ranking score itself. It's fewer hallucinations reaching the user.

When reranking is not worth adding

Reranking isn't free, and it isn't automatically a win. Adding a second stage adds latency, adds infrastructure, and in some setups adds nothing measurable. The default assumption should be "prove it helps," not "add it and move on," and teams that skip that proof usually find out the hard way, six months in, that the reranker was never doing anything.

A few questions are worth asking honestly before wiring one in. Are users actually seeing irrelevant or hallucinated answers even though recall looks fine? If the right document was never retrieved in the first place, that's a recall failure, and reranking can't fix it, it only reorders what's already there. Is top-5 precision meaningfully worse than top-50 recall? If that gap is small, the ranking step is probably already doing its job well enough. And does the corpus even have enough scale to produce a ranking problem to begin with? Corpora under roughly 5,000 documents often don't, and running a reranker on a small, clean corpus is mostly just burning milliseconds for show.

Recent research backs skepticism toward a fixed, one-size-fits-all rerank step. DynamicRAG, from 2025, proposes an agent that decides on its own how many documents to rerank and in what order, instead of always reranking a fixed top-100. That's a direct challenge to the "always rerank the top 100" default most pipelines run on autopilot without ever questioning. A separate paper on multi-criteria reranking, from March 2025, argues relevance alone isn't a sufficient signal and proposes evaluating retrieval across multiple relevance criteria at once, another sign the field is moving past single-score relevance as the whole story.

Skip reranking for small corpora, narrow-domain pipelines with heavy keyword overlap, and generators that already tolerate a messy candidate order fairly well. The real calculation is a trade, every time: weigh the cost of a bad answer (a hallucinated fact, a missed clause, a wrong recommendation) against the 50 to 150 milliseconds and the compute the reranker adds. Sometimes that trade doesn't clear, and forcing a reranker into a pipeline that doesn't need one just burns latency for nothing.

Latency, cost, and routing in production reranking pipelines

Budget for reranking has to include its neighbors, not just itself. Cross-encoder reranking adds 50 to 150 milliseconds on its own. Query rewriting, a common step that often runs right before it, adds another 200 to 400 milliseconds. Stack both, and the latency budget for a single query response gets tight fast.

On CPU, a MiniLM-class model scoring 50 candidates runs somewhere around 100 to 300 milliseconds, depending on document length. Move that to a single GPU and it drops under 50 milliseconds. The throughput ceiling from earlier is worth repeating, because it's the number that actually forces the architecture decision: at 40 queries per second, a standard cross-encoder's p99.9 latency can blow past 21 seconds. That's the wall that pushes high-throughput systems toward ColBERT or a lighter model class instead, and no amount of GPU spend fixes that math on its own.

A handful of fixes show up again and again in production systems. Caching helps on both ends: query rewrites cache cheaply, and reranker outputs for repeated queries cache the same way. Managed, GPU-backed APIs like Cohere and Voyage take the infrastructure question off the table entirely, predictable latency without anyone provisioning a GPU cluster. ANN indexes such as HNSW or IVF-PQ at the retrieval stage shrink the candidate pool before it ever reaches the reranker, cutting the reranker's workload directly. And for pipelines that don't need a synchronous response in real time, batching on CPU with an async pipeline is often good enough.

Routing by query type is another lever worth pulling. A small classifier can sort incoming queries by intent: code queries go to a code-capable reranker, multilingual queries go to a multilingual model, plain English queries go to something lightweight. That keeps cost down without sacrificing accuracy on the queries that actually need the heavier model.

Two research directions worth tracking on the latency-efficiency front. CROSS-JEM reports 4x lower latency than standard cross-encoders while holding accuracy steady. A SIGIR 2025 paper on early-exit cross-encoders introduces mechanisms to cut inference cost without giving up accuracy. Both point the same direction: toward cross-encoders that stay accurate but stop being so heavy to run.

One more distinction matters here. Pipelines feeding agents asynchronously, rather than answering a user in real time, run under much looser latency constraints, and that opens the door to heavier, more accurate models that would never survive a synchronous, user-facing SLA.

Wiring reranking into a RAG pipeline with live web data

Reranking can only sort what's already in front of it. A perfectly ranked set of stale, outdated chunks still produces a wrong answer, just a confidently wrong one, sitting at the top of the list.

Go back to the deprecated-API example from the start. The correct docs page existed in the index, sitting at position 23. Reranking is what promotes it to where the LLM can see it. But if that index had come from a live crawl instead of running on stale data, the outdated chunk that confused the model in the first place might never have made it into the candidate pool at all. Reranking fixes ordering, it does nothing for freshness. Treating it as a fix for both is exactly the mistake that keeps deprecated docs surfacing in production long after they've been retired.

That's the part of the pipeline that's easy to overlook: a RAG system is only as good as the freshness and cleanliness of what it indexes. No amount of reranking sophistication makes up for a corpus that no longer reflects the current state of the web. A growing share of the web scraping happening in 2026 exists specifically to feed LLM pipelines, RAG systems, and autonomous agents, and that puts real pressure on how that content gets collected and kept current.

Building and maintaining scrapers just to keep a RAG index fresh is the kind of work that eats engineering time without ever showing up as a feature. In the old model, most of that effort goes into upkeep: patching selectors, handling site redesigns, chasing layout changes instead of building anything new. Research from McGill University in 2025 found that AI-based extraction methods held 98.4% accuracy even as page structures changed, and setup time dropped from weeks down to hours (per Kadoa, 2026). That's a solid illustration of why purpose-built crawling infrastructure tends to beat hand-rolled scrapers when the job is keeping an index current, not just building one once.

Format matters just as much as freshness. Clean, token-efficient Markdown from a crawl is what both rerankers and LLMs are built to read. Raw HTML drags in markup noise, navigation clutter, and boilerplate that hurts retrieval quality before reranking ever gets a chance to help, and that same noise drags down reranking quality too. For agentic RAG, where an agent triggers retrieval on demand instead of working off a fixed, pre-built index, the same kind of on-demand URL-to-Markdown conversion can feed live pages straight into the pipeline without needing a static index sitting behind it at all.