Hybrid Search With Dense and Sparse Vectors in RAG

A user searches "SKU AZ-4471" and the dense retriever hands back a similar-sounding product that isn't it. The exact document sits right there in the index. It just never surfaces, because the embedding model never learned that string as anything special. Flip the scenario: someone asks "how do I return a damaged item?" and the knowledge base has an article titled "Refund Policy for Defective Products." Zero word overlap. BM25 comes back empty-handed. These aren't rare glitches you patch around later. They're the two most common ways retrieval quality dies in production RAG systems, and they happen for opposite reasons: dense models miss exact strings because they encode meaning, not characters, and BM25 misses meaning because it only knows exact strings. That split is exactly why hybrid retrieval, done right, closes both gaps in a single system.

Hybrid search in RAG combines at least two retrieval signals, typically a dense vector retriever and a sparse lexical retriever like BM25, then merges their ranked outputs into a single candidate set before passing results to the language model. Dense retrieval handles semantic matching, surfacing documents that share meaning with the query even when no words overlap, while BM25 handles exact-match retrieval, catching SKUs, error codes, version strings, and any term the embedding model never learned to represent well. The two methods fail in opposite directions, which is what makes combining them structurally useful rather than merely additive. Pure vector retrieval is the right default only when queries are consistently paraphrastic and the corpus contains no rare identifiers; pure BM25 is reasonable only when queries are consistently keyword-shaped and synonyms never matter. In any system where both query types show up, which is most production RAG systems, hybrid retrieval closes both gaps in a single pipeline.

What BM25 actually does, and where it breaks

BM25 runs on an inverted index. For every term in the corpus, it keeps a list of which documents contain that term and how often. Query time is just a lookup and a score, no neural network involved.

Two mechanisms separate it from plain TF-IDF, and both matter more than people give them credit for. Term frequency saturation means the tenth occurrence of a word in a document adds a lot less to the score than the second occurrence did, so nobody can game the ranking by stuffing keywords. Document length normalization penalizes long documents so a 40-page manual doesn't automatically outrank a 200-word FAQ just because it contains more words overall.

Two parameters control this, and they've got standard defaults for a reason: k1 = 1.2 governs how much repeated terms matter (push it toward 2.0 and repetition counts for more), and b = 0.75 governs the length penalty (1.0 is full penalty, 0 turns it off entirely).

BM25 is excellent at exact-match retrieval: SKUs, error codes, version numbers, legal clause references, person names. It's also good at catching terms an embedding model has genuinely never seen. And operationally, it's cheap: no GPU, no inference at query time, just index lookups on CPU. That's a real advantage at scale.

Where it falls apart is anything requiring understanding rather than matching. Synonyms, paraphrases, conceptual questions like "how to cancel" against a document titled "subscription termination process," anything crossing languages. BM25 doesn't know "cancel" and "terminate" are related ideas. It just doesn't see the connection at all.

What dense embeddings capture that BM25 cannot, and where they fail in turn

Dense retrieval runs both the query and every document chunk through a transformer encoder, producing vectors somewhere in the hundreds to over a thousand dimensions, every coordinate a real number. At query time, an approximate nearest neighbor index (HNSW is the dominant approach, used by both Qdrant and Weaviate; Pinecone instead runs its own proprietary indexing architecture) finds documents whose vectors sit close to the query vector by cosine similarity or dot product.

The strength here is real semantic matching. Search "automobile" and you'll pull documents about "cars" and "vehicles," because their vectors land near each other in the embedding space even though the words share no letters. Open-source model families like Sentence-BERT, E5, and BGE handle this well, and hosted options from OpenAI, Cohere, and Voyage AI do the same thing as an API call.

The failure mode mirrors BM25's strength almost exactly. Rare identifiers, product codes, error strings, version numbers: if the embedding model never saw enough of these during training, they land somewhere generic in vector space, nowhere near where they should be. Late-interaction approaches like ColBERT chip away at this by matching at the token level instead of compressing everything into one vector, but that costs a lot more compute per query.

There's a cost story here too. GPU-backed indexing carries substantially higher infrastructure overhead than an inverted index, and swapping embedding models means reindexing the entire corpus from scratch, offline, before anything works again. Worth sitting with the dimensionality contrast for a second: dense vectors run hundreds to over a thousand dimensions, every one of them a nonzero float. Sparse vectors run many thousands of dimensions, and almost all of them are zero. That single fact explains most of the storage and compute tradeoffs between the two approaches.

SPLADE and learned sparse models: the third retrieval primitive developers often overlook

SPLADE sits in a strange middle ground, and most developers building RAG pipelines have never heard of it. It uses masked language modeling, the same underlying technique behind BERT, to produce a sparse vector, but the weights it assigns aren't raw term counts like BM25 uses. They're learned.

Take the word "automobile" again. A SPLADE document representation puts real weight not just on "automobile" itself but on "car," "vehicle," and "motor," terms that never appeared in the original text at all. That's BM25-style sparsity (most dimensions still zero) fused with something closer to the semantic reach of a dense model.

This isn't a variant of BM25. It's trained end-to-end, and that training is what lets it close synonym gaps that BM25 structurally cannot, while keeping the lexical precision that made BM25 useful in the first place. On standard benchmark evaluations, SPLADE has shown strong gains over traditional BM25. Qdrant and Pinecone both support it as a sparse component in hybrid pipelines now, alongside BM25.

The clearest evidence for why this matters comes from Ant Group's production RAG evaluation. Starting from an existing BM25 plus dense baseline, adding SPLADE-style sparse vectors pushed Recall@3 from 55.60% to 74.10%, an 18.5-point jump, and Recall@10 from 85.20% to 88.90%. That's not a marginal tuning gain. That's a third signal doing real work on top of two signals that were already in place.

The catch: SPLADE needs inference at index time, so it's slower to build than a BM25 index. It's not something every pipeline needs on day one. But when BM25 keeps missing synonym-heavy queries and reindexing cost isn't the bottleneck, it's the upgrade worth making.

How Reciprocal Rank Fusion solves the score-incompatibility problem that breaks naïve weighted averaging

Here's the problem nobody notices until it bites them: BM25 scores are unbounded positive numbers, they can be 4, 40, or 400 depending on the corpus. Cosine similarity from a dense model is bounded between -1 and 1. Average those two directly and BM25 dominates by sheer numerical gravity, not because it's actually more relevant. Normalization schemes exist, but none of them hold up reliably across different query distributions.

Reciprocal Rank Fusion sidesteps the whole mess by throwing out scores entirely and working with rank position instead. The formula: score(d) = Σ 1/(k + rank_i(d)), summed across every retriever the document appears in, with k = 60 as a smoothing constant.

The logic is intuitive once you see it. A document that lands near the top of both the BM25 ranking and the dense ranking accumulates a high combined score, because it's getting a strong contribution from both lists. A document that only one retriever likes gets a much smaller contribution. Consensus wins, not raw magnitude.

That k = 60 constant isn't arbitrary. It comes from the original Cormack et al. SIGIR paper in 2009, and it's held up remarkably well since. Lower k values give more weight to whatever's sitting at rank 1; higher k flattens things out so the whole list matters more evenly.

This is exactly why RRF handles the failure modes from the opening. If the dense retriever surfaces something plausible but wrong, and BM25 surfaces the actual SKU match, RRF rewards the document with support across both lists rather than trusting whichever retriever sounds more confident. No normalization step needed, no tuning required to beat an untuned weighted average. That's the whole operational case for RRF.

One caveat worth flagging directly: teams on Weaviate 1.24 who upgrade without explicitly setting fusionType get silently switched from RRF (called rankedFusion there) to Relative Score Fusion. Different algorithm, different result ordering, no warning banner. Pin fusionType explicitly if there's a baseline evaluation worth protecting.

Weighted alpha fusion as an alternative to RRF, and when the choice matters

Weighted alpha fusion is the more direct approach: score(d) = α × dense_score + (1 − α) × sparse_score. Weaviate defaults to α = 0.75, meaning a strong lean toward dense over BM25, and LlamaIndex exposes alpha as a tunable parameter for anyone who wants to hand-adjust the balance.

The catch is that this method requires normalizing the two score distributions before combining them, and that normalization is sensitive to shifts in query distribution that are hard to predict ahead of time. What works for this month's traffic might not hold for next month's.

Where weighted fusion earns its place: a stable, well-understood query distribution where explicit control over the dense-versus-sparse balance actually matters. A catalog-heavy system where exact-match SKU and model-number queries dominate is the textbook case. A reasonable starting point there is BM25 at 0.7 and dense at 0.3, then tune from that baseline against a held-out evaluation set.

RRF earns its place in the opposite situation: mixed or unpredictable query traffic, multi-system pipelines where score scales don't line up cleanly, or any case where the complexity of normalization outweighs the benefit of fine-grained weight control.

Worth being precise about where the field actually stands on "smarter" fusion. Weaviate's hybrid search, as of 2025, fuses BM25 and vector search through a tunable alpha parameter and a choice between two fusion algorithms, rankedFusion or relativeScoreFusion. Learned, dynamic fusion weights trained from labeled data represent a theoretical ceiling for accuracy beyond what current documented offerings describe. The fusion itself combines scores from a vector search pass and a BM25 pass, coordinated at query time. No "Weaviate 2.0" or unified-index release has shipped.

What do the benchmarks show about hybrid versus pure vector or keyword retrieval?

Start with the WANDS e-commerce benchmark, because it's concrete and the setup matches the opening scenarios almost exactly. Hybrid retrieval using RRF scores 0.7497 NDCG, against 0.6983 for BM25 alone and 0.6953 for dense alone (Turnbull, 2025). That's a 7.4% lift over either method run solo, in a catalog setting with both exact SKUs and descriptive product language mixed together.

Move to financial documents, a domain full of mixed prose and tables, exactly the kind of vocabulary mismatch BM25 exists to solve. On financial documents, hybrid retrieval combined with neural reranking has been reported hitting Recall@5 of 0.816, against 0.587 for dense-only. That's not a marginal improvement, that's a different tier of retrieval quality.

Then there's the number that demands a second look: a high-recall benchmark from Weinberg (2026, ResearchGate preprint) shows simple weighted fusion at α = 0.5 delivering up to a 580% improvement in Recall@10 over dense-only on MS MARCO, jumping from 13.9% to 80.8%. That figure looks almost too dramatic to trust, until you notice the baseline. Dense retrieval sitting at 13.9% Recall@10 is a genuinely weak starting point, reflecting dense models' known soft spot: they're better at ranking a handful of good answers near the top than at finding every relevant document in the corpus. Hybrid's 80.8% isn't an incremental bump on top of that weakness, it's a shift into an entirely different recall regime.

Taken together, BEIR, MTEB, WANDS, and MS MARCO all point the same direction: hybrid retrieval isn't a niche fix for one domain, it's a structural improvement that shows up wherever both lexical and semantic queries coexist. That said, none of these numbers were measured on any particular team's corpus or query mix. The honest takeaway isn't "expect a 7.4% lift" or "expect 580%." It's start with hybrid as the default architecture, then measure against an actual eval set built from real queries.

Cross-encoder reranking as the second-stage precision layer on top of hybrid retrieval

Worth being exact about what each piece of this pipeline actually does, because conflating them is the most common architectural mistake in RAG systems that underperform in production. BM25 and dense search are first-stage retrievers: they scan the whole corpus and produce a candidate set. RRF is a fusion algorithm that merges the outputs of those retrievers. Cross-encoder reranking is a different operation entirely: a second-stage precision pass over a shortlist, typically somewhere in the range of hundreds to a few thousand candidates.

Cross-encoders can't replace first-stage retrieval, and that's not a limitation to work around, it's just how the math works. A cross-encoder computes a joint relevance score for a single query-document pair at a time. Running that computation against millions of documents at query time isn't slow, it's simply infeasible. It needs a shortlist handed to it first.

What it adds on top of that shortlist is real. Per BEIR and MTEB evaluation literature, a cross-encoder reranker layered on a hybrid first stage typically adds another 5 to 15 points of MRR on hard evaluation sets, exactly the queries where first-stage retrieval alone struggles to separate a good answer from a nearly-good one.

Managed options make this easy to bolt on without hosting anything: Cohere Rerank 3.5, Voyage AI's rerank-2.5, and the lighter rerank-2.5-lite all run as REST APIs. Voyage's rerank-2.5, released in August 2025, added something newer: instructions written in plain language, attached to the reranking call, that steer what counts as relevant for that specific query. Voyage reports a 7.94% accuracy improvement over Cohere Rerank v3.5 across a 93-dataset suite, worth noting that's a vendor-reported figure, not an independent benchmark.

The mental model worth keeping: BM25 plus dense retrieval, then RRF fusion, then cross-encoder reranking. Three distinct stages, each one narrowing the candidate set and sharpening precision. Trying to collapse all three into a single scoring formula is the shortcut that breaks in production.

Where the major vector database platforms sit on hybrid search in 2026

Qdrant runs native RRF through its query interface, added server-side in v1.10, and supports SPLADE-style learned sparse vectors as the sparse half of the pipeline alongside plain BM25.

Weaviate offers a selectable fusionType, rankedFusion for RRF or Relative Score Fusion as the alternative. Hybrid Search 2.0, shipped in October 2025, adds learned fusion on top of a unified vector-plus-keyword index. Separately, Weaviate restructured its pricing in late 2025, moving Serverless into a Shared Cloud tier and replacing the old Serverless and Enterprise plans with new Flex, Plus, and Premium tiers, alongside AI-native Agents now available across Cloud plans.

Pinecone runs alpha-weighted dense and sparse fusion inside a single index. Its serverless offering has become its primary recommended deployment path.

Elasticsearch offers native RRF on its Enterprise tier. Teams on lower tiers aren't locked out entirely, there's a client-side workaround using the ranx Python library. Amazon OpenSearch ships native hybrid support as well.

Line up all five platforms and the pattern is hard to miss: Weaviate, Qdrant, Pinecone, Elasticsearch, and OpenSearch have each built the same architectural pattern into their core product. That's not five vendors chasing a research trend. That's hybrid retrieval becoming the thing developers reach for by default, the same way nobody debates whether to use an index on a database column anymore.