Hybrid Search in RAG: BM25 + Semantic Search in Python

Your RAG app has been solid for weeks. Then someone pastes in E2077, a code that appears on exactly one page of your knowledge base, and the retriever hands back three other error pages. Hybrid search exists for exactly that failure, and it takes about seventy lines of Python.

Here's what we're building. A 22-page support knowledge base goes in one end. Two independent retrievers rank every page: BM25 counts exact word matches and rewards rare terms, while a vector search compares dense embeddings and matches meaning instead of spelling.

Each retriever hands back its own ranked list. The two score on completely incompatible scales — BM25 is unbounded, cosine similarity is capped at 1.0 — so you can't just add the numbers. Reciprocal Rank Fusion sidesteps that by discarding the scores entirely and merging on rank position alone.

The last stage is the one most tutorials skip: a labelled query set and three metrics. They tell you whether fusion actually helped, or whether you just doubled your latency for nothing.

Prerequisites: Python 3.9+, numpy 1.24+, and familiarity with chunking for RAG. Everything on this page runs in your browser — no API key, no vector database, nothing to install.

Hybrid search runs two different retrievers over the same corpus and merges their ranked results into one list. In a RAG pipeline that almost always means a keyword retriever (BM25) alongside a dense vector retriever (embeddings plus cosine similarity).

The reason to bother isn't that one is better than the other — it's that they fail in different places. BM25 is blind to wording it has never indexed. Vector search is blind to rare exact strings: product codes, error codes, SKUs, function names, surnames. Put them together and each covers the other's blind spot.

We need a corpus before we need retrievers. The one below is a miniature support knowledge base for a fictional file-sync product called Nimbus, and it has one deliberate quirk that makes it a good test bed.

The first six are error-code pages, written the way real ones are: near-identical boilerplate. They differ only in the code itself and one sentence about the cause. The other sixteen are concept pages about tokens, throughput, billing, filters and so on. Run this to load the lot.

Twenty-two pages averaging about forty words each. Small enough to inspect by hand, big enough that a retriever has to actually choose.

An embedding model never sees your text as words. It sees subword pieces. A tokenizer chops every string into fragments from a fixed vocabulary, and anything rare — E2077, TS2304, ERR_CONN_RESET — shatters into pieces that carry no meaning of their own.

Then the model averages those pieces into one fixed-size vector. That vector has to describe the whole passage in, say, 384 or 1,536 numbers. A single rare token that appears once barely moves it.

Two functions make this concrete. keyword_tokens is what a keyword index stores: whole words, lowercased, minus a short stop-word list. embedding_tokens stands in for an embedding model's tokenizer. It shatters anything containing digits into short fragments, and it collapses word endings so copies, copied and copying all land on the same piece.

Watch what each one does to the same sentence.

Look at the last three lines. E2045 becomes ['e', '20', '45'] and E2077 becomes ['e', '20', '77']. They share two pieces out of three. To the keyword index they're two unrelated terms; to the embedding pipeline they're near-twins.

That isn't an artefact of our toy tokenizer. It's the everyday behaviour of BPE and WordPiece tokenizers on strings they have never seen.

Building the vectors takes three steps. First, count subword pieces per page and weight them by inverse document frequency, so common pieces count for less. Second, run a singular value decomposition over that sparse matrix and keep only the top 10 directions — that's the compression step, and it's where rare pieces get flattened. Third, expose one embed() function that turns any string, page or query, into the same 10 numbers.

Three hundred and sixteen sparse pieces squeezed into ten numbers per page. Those ten numbers mean nothing on their own. Only the angle between two of them carries information, which is why the next step measures cosine similarity rather than distance.

Now the search function, and the moment of truth. We ask a question containing a literal error code and see which pages come back.

Predict the ranking before you hit Run. The query says E2077, and exactly one page in the corpus contains that string. Where does it land?

The right page comes back third. E1120 wins and E3110 takes second. The page that literally contains the string the user typed limps in at 0.88, indistinguishable from the two wrong answers above it.

If your RAG prompt stuffs the top 3 chunks, you got lucky. If it stuffs the top 1 — common when chunks are large — you just answered a question about a full disk with a page about an unreachable host.

BM25 scores a document against a query by adding up one number per query term. That per-term number answers three questions.

How rare is this term across the whole corpus? How often does it appear in this document? And is the document long enough that the repetition might be an accident?

Take them one at a time, starting with rarity. Inverse document frequency (IDF) runs high for terms that appear in few documents and low for terms that appear everywhere. I think the formula is worth reading once before the code, because every later surprise comes out of it:

The 1 + inside the logarithm matters far more than it looks, and we come back to it in the mistakes section. For now, watch how the score collapses as a term gets more common.

The term e2077 is worth almost seven times what job is worth. That single ratio is why BM25 finds error codes and embeddings don't. Rarity is the whole point of the scoring function, not a rounding error inside it.

Next, term frequency. The naive approach is to count occurrences, but that rewards keyword-stuffed pages absurdly. A page saying checksum fifty times isn't fifty times more relevant than a page saying it once. So BM25 puts the count through a saturating curve controlled by k1, then divides by a length penalty controlled by b.

The next block prints that curve for a single term at six frequencies, across three document lengths: average, half-length and double-length. Two things to watch for. Going from one occurrence to fifty barely more than doubles the score, and the same frequency in a document twice as long is worth noticeably less.

One occurrence in an average-length document scores exactly 1.000, which makes the table easy to read as a multiplier. Fifty occurrences buys you 2.427. That's a hair under two and a half times the value of a single mention, for fifty times the repetition.

Assembling the full ranker is now just a loop: for each query term, multiply its IDF by its TF component in every document and accumulate. Same query as before — the one where vector search buried the right page at rank three.

E2077 at rank 1 with 4.073, and the runner-up scores 1.248. That isn't a narrow win, it's a landslide. The rare term contributed almost all of the score, and only one document has it.

So keyword search wins this round decisively. Hold that thought, because the next query flips the result completely.

You now have two retrievers that disagree, and the obvious move is to combine their scores. The obvious move doesn't work. It's worth seeing exactly why before reaching for the fix.

Here's a query where the two retrievers agree on the winner but disagree about almost everything below it. The block prints both ranked lists side by side so you can compare the raw numbers.

BM25 tops out at 5.771 and cosine tops out at 0.83. Those aren't two measurements of one quantity in different units. BM25 has no upper bound at all, and its magnitude depends on how many query terms happened to be rare.

Add them together and the outcome is predictable: whichever number is bigger decides the ranking. The next block does exactly that, printing the naive merge next to BM25 on its own.

One swap between positions two and three. That's the entire contribution of the semantic retriever — its scores were an order of magnitude too small to matter. You built two retrievers and shipped one.

The standard repair is min-max normalisation: rescale each list so its best hit becomes 1.0 and its worst becomes 0.0, then blend with a weight alpha. Setting alpha=1.0 gives you pure BM25, alpha=0.0 gives pure semantic, and anything in between mixes them.

That works. At alpha=0.0 the ranking is pure semantic and E2077 sits third; anywhere from 0.3 to 0.7 it's first. The blend genuinely rescued the query.

But min-max has a flaw that never shows up on a single query, and it's the reason the next section exists. The rescaling happens per query, so the top hit always becomes 1.0 — whether it was a screaming match or a shrug.

Reciprocal Rank Fusion throws the scores away. Each retriever votes with positions instead: a document at rank r earns 1 / (k + r) points, the points are summed across retrievers, and the totals become the final ranking.

A rank is a rank no matter which retriever produced it, so there's nothing to normalise and nothing to tune. That is why RRF is my default recommendation: it has no knobs, so there is nothing to get wrong. Cormack, Clarke and Buettcher introduced it in 2009 and reported that it outperformed the individual systems it combined, plus several learned combination methods, without any training data.

The implementation is about eight lines. reciprocal_rank_fusion takes ranked lists of document ids and returns the merged ranking; hybrid_search wires our two retrievers into it. Watch the depth parameter — you fuse the top 10 from each side, then return the top 5.

Two queries, two opposite failures, one fix. On the error-code query BM25 was right and the vector side was wrong, and the hybrid list opens with E2077. On the paraphrase query BM25 latched onto common words and put conflicts first, while the vector side got webhooks — and the hybrid list opens with webhooks.

That leaves rrf_k, which controls how steeply the points fall off with rank. At rrf_k=0 the top hit is worth ten times the tenth hit; at 60 it's worth 1.15 times. Small k means the top of each list dominates. Large k means deep agreement counts almost as much as a first-place vote.

Honest caveat: on a corpus this small, sweeping rrf_k from 1 to 200 changes nothing at all. Verify that yourself with the benchmark below. It starts to matter when your lists are long and the two retrievers disagree deep down — a real situation at 100k documents, and a fictional one at 22.

Anecdotes about two queries prove nothing. To know whether hybrid search earns its extra latency, you need a labelled query set: real questions, each tagged with the page that should come back. You also need metrics that summarise many queries at once.

We'll use three. Recall@1 is the fraction of queries whose correct page came back first. Recall@3 is the fraction whose correct page appeared anywhere in the top three. MRR (mean reciprocal rank) averages 1 / position across queries, so rank 1 scores 1.0, rank 2 scores 0.5, and a miss scores 0.

The query set below is sixteen questions in two deliberate halves. Eight are exact queries containing a code, a command, an HTTP status or a version string. Eight are natural queries phrased the way a frustrated user actually types. A real support inbox holds both, in roughly that proportion.

Now the scoring harness. evaluate takes any search function with the signature fn(query, k), runs the whole query set through it, and returns the three metrics. Four engines go in: BM25 alone, vectors alone, weighted fusion at alpha=0.5, and RRF.

The two single retrievers are dead level on aggregate — 81% recall@1, MRR within 0.002 of each other. Look only at that summary row and you'd conclude they're interchangeable, pick one, and ship.

They're not interchangeable at all, and the split by query type makes that obvious. Run this next.

On the eight exact queries BM25 scores a perfect MRR of 1.000 and the semantic retriever manages 0.917. On the eight natural queries the order reverses: BM25 drops to 0.775 while the semantic retriever holds 0.854. Two mirror-image profiles, and an aggregate score that hides both.

Both hybrids score a perfect 1.000 on the exact half and 0.854 or better on the natural half. That's the actual value proposition: you stop having to guess which kind of query your users will send.

Hybrid search costs you a second index to build, a second query on every request, and a fusion step to maintain. Sometimes that buys nothing at all. Here's the clearest case: a query whose words appear nowhere in the corpus.

Not one query term exists in the index. Both retrievers return arbitrary pages, and the hybrid dutifully returns an arbitrary page too — scored, ranked first, looking exactly as authoritative as a correct answer. Fusing noise with noise gives you noise with better presentation.

The fix for that query isn't fusion. It's a synonym list, a query-rewriting step, or writing the documentation that's missing. Skip hybrid search entirely when any of these describe your situation:

You won't ship the code above. You'll ship a maintained BM25 implementation, a real embedding model, and probably a vector database that does the fusion for you. The two blocks below are the drop-in replacements.

Neither is runnable here. rank_bm25, sentence-transformers and langchain aren't available in the in-browser Python runtime, and the encoder would need to download model weights. Notice that the RRF function is identical to the one you already have — that's the point.

LangChain wraps the same idea in EnsembleRetriever, which is worth knowing because it's what most teams reach for first. Note that it does weighted RRF, not plain RRF — the weights argument scales each retriever's reciprocal-rank points before summing.

If you're already on a managed vector store, I'd check its feature list before building any of this. Qdrant, Weaviate, Elasticsearch, OpenSearch and pgvector-based stacks all expose hybrid retrieval with RRF as a server-side feature. That saves you a round trip and keeps the two indexes in sync.

This one produces no error and no warning. The index stores lowercased terms, the query tokenizer forgets to lowercase, and every search for a capitalised code silently returns nothing.

Fix: call the exact same tokenizer function on queries and on documents. Not an equivalent one — the same one, imported from the same place. Add stemming or a stop-word list later, and both sides change together for free.

The BM25 paper defines IDF as log((N - df + 0.5) / (df + 0.5)), with no 1 +. For any term appearing in more than half your documents, that expression goes negative. A negative IDF means every extra occurrence actively pushes the document down the ranking.

job appears in 15 of 22 pages and scores -0.726 under the classic formula. A page mentioning job five times gets demoted for it.

Fix: use the smoothed variant log(1 + (N - df + 0.5) / (df + 0.5)), which is what Lucene, Elasticsearch and rank_bm25 all ship. It's always positive, and it decays gracefully to near-zero for terms that carry no information.

You saw this one in action earlier. BM25 is unbounded and typically lands between 0 and 10; cosine similarity is squeezed into 0 to 1. Summing them isn't fusion, it's BM25 with a rounding error attached.

If you do go with weighted fusion, the textbook min-max line crashes the moment a retriever returns a flat list. That happens more often than you'd think — any query whose terms are all missing from the index gives BM25 a column of zeros.

Fix: return all zeros when hi == lo, as our min_max does. Zero is the honest answer: a retriever that scored everything identically expressed no preference, so it should contribute none to the blend. RRF sidesteps the whole problem, which is one more reason to prefer it.

Everything above, condensed into one block: both tokenizers, BM25, the embedding model, and RRF. Paste it straight under the corpus cell from the top of this page and you have a working hybrid retriever in about seventy lines.

Retrieve roughly three to five times the number of results you actually want, from each side. If your prompt takes the top 5 chunks, pull 20 from BM25 and 20 from the vector index, fuse, then slice the top 5. Going deeper costs almost nothing for BM25 and nothing at all for a vector index that already scored everything. It also gives RRF room to promote pages that neither retriever ranked first.

They solve different problems and compose well. Hybrid search widens the candidate pool so the right chunk is present; a cross-encoder reranker reorders that pool so the right chunk is first. The usual production shape is hybrid retrieval at depth 50, then a reranker down to the top 5. If you can only afford one, start with hybrid — a reranker can't promote a document that was never retrieved.

Better embeddings shrink the gap but don't close it, because the failure is structural. Any model that compresses text into a fixed-size vector has to discard something, and rare exact strings are the cheapest thing to discard. If your users paste identifiers — error codes, order numbers, part numbers, function names — keep the keyword index no matter how good the encoder gets.

Learned sparse retrievers such as SPLADE sit between the two approaches. They produce sparse term weights like BM25, but expand each document with related terms the way an embedding model would. They usually beat BM25 on benchmarks, at the cost of a model inference per document at index time and per query at search time. BM25 is still the right first move: no model, no GPU, and no retraining when your corpus changes.

Deduplicate before fusing, not after. RRF keys on the document id, so the same chunk stored under two ids collects two separate piles of points and can outrank a genuinely better result. Give every chunk one stable id — a hash of its text works fine — and use that id in both indexes.