BM25: Lexical Ranking
BM25 (Best Matching 25, from the Okapi project) is the ranking function that has powered keyword search for decades. It is the default in Lucene, Elasticsearch, and OpenSearch, and the "sparse" half of most hybrid RAG retrievers. It scores a document against a query using only term overlap: no embeddings, no training, no neural network. Given a query, it ranks every document by a sum of per-term weights that reward matching rare words, reward repeated matches with diminishing returns, and correct for document length.
It is worth understanding precisely because it is an absurdly strong baseline. A well-tuned BM25 index often matches or beats a dense retriever on exact-match queries (names, error codes, IDs, rare jargon) at a tiny fraction of the cost, which is exactly why production systems fuse the two.
From TF-IDF to BM25
Classic TF-IDF scores a query term in document as — raw term frequency times inverse document frequency. Two problems:
- Term frequency grows linearly and unboundedly. A document that says "cat" 100 times looks 100× more relevant than one that says it once. In reality, relevance saturates — the 10th mention adds little over the 2nd.
- Long documents win by accident. A long document repeats every term more often, so it out-scores a short, focused one for no good reason.
BM25 fixes both with two ideas: a saturation curve on term frequency, and an explicit length-normalization knob.
The formula
For a query and document :
where
- — the raw count of term in document ,
- — the document length in words, and — the average document length across the corpus,
- — the saturation parameter (typically –),
- — the length-normalization parameter (typically ),
- — the inverse document frequency of the term (below).
The IDF uses the probabilistic (Robertson–Spärck-Jones) form, with documents and containing the term:
The inside the log keeps IDF non-negative even for very common terms (a term in more than half the corpus would otherwise go negative).
Reading the two knobs
: term-frequency saturation. Look at the fraction: as , the term score approaches its ceiling . Small saturates fast (presence ≈ enough); large keeps rewarding repeats. At the term frequency drops out entirely and BM25 degenerates to a pure binary IDF score.
Why saturation matters
The saturation curve is BM25's defining feature. The first occurrence of a rare query term should move the ranking a lot; the twentieth should barely move it. A linear TF (plain TF-IDF) lets keyword-stuffed documents dominate — BM25's diminishing returns make it robust to that.
: length normalization. The denominator scales by . At length is ignored entirely; at scores are fully normalized by relative length, so a document twice as long needs twice as many matches to score the same. is the standard compromise — penalize length, but not brutally.
A worked score
Corpus of documents, words. Query: "neural retrieval". Take a document with words containing "neural" 3× and "retrieval" 1×. Say "neural" appears in docs and "retrieval" in . Use , .
First the IDFs:
The length factor is , so .
"neural" ():
"retrieval" ():
Notice the single-occurrence rare term ("retrieval") contributes more than three occurrences of the commoner one ("neural") — saturation plus IDF working together. That is the behavior a linear TF-IDF would get wrong.
Watch it rank
One document scored by hand shows the formula; ranking a corpus shows what the formula is for. Step through BM25 scoring five documents for the query "neural retrieval" — IDF first, then each document term by term, then the final ordering. (This corpus is its own worked example, so its , and IDFs differ from the hand-worked numbers above.)
BM25 — scoring and ranking a corpus, term by term
Five short documents, query "neural retrieval". BM25 needs only three corpus statistics: the number of documents N = 5, the average document length avgdl = 6.8, and each term's document frequency. No training, no embeddings.
| doc | |d| | neural | retrieval | score |
|---|---|---|---|---|
| d1 | 5 | 1 | 1 | — |
| d2 | 5 | 2 | 0 | — |
| d3 | 13 | 1 | 2 | — |
| d4 | 5 | 1 | 0 | — |
| d5 | 6 | 0 | 0 | — |
Note d3: it contains the query terms most often (neural ×1, retrieval ×2) but is by far the longest document. Watch whether that wins or loses.
Watch BM25 rank five documents for the query "neural retrieval". The two ideas BM25 adds to TF-IDF are visible in the numbers: IDF makes one match on the rare "retrieval" outweigh two matches on the common "neural", and the length factor drops the long d3 below the short d1 despite d3 matching more terms. The last step re-runs everything with b = 0 so you can see the long document win the moment length normalization is removed.
Both of BM25's corrections change the answer here, and you can watch each one do it:
- IDF. "neural" appears in 4 of 5 documents and "retrieval" in 2, so one
match on "retrieval" is worth roughly 3× one match on "neural".
d1's single rare match beatsd2's two common ones. - Length normalization.
d3contains more query-term occurrences thand1but is 2.6× longer, so its length factor exceeds 1 and shrinks every contribution — it lands second. The final step re-scores the same corpus with andd3jumps to first, which is exactly the keyword-stuffing failure is there to prevent.
BM25 vs dense embeddings
| BM25 (lexical / sparse) | Embeddings (semantic / dense) | |
|---|---|---|
| Matches | exact tokens (& stems) | meaning / paraphrase |
| "car" vs "automobile" | miss (no shared token) | hit |
| Rare IDs, codes, names | excellent | often poor |
| Training needed | none | embedding model |
| Index | inverted index | ANN vector index |
| Cost | very cheap | embedding + ANN |
| Out-of-domain terms | robust | can drift |
Neither dominates. Embeddings win on synonymy and paraphrase; BM25 wins on precise tokens and unseen jargon, with zero training and near-zero cost. This is why hybrid search fuses both scores — commonly with Reciprocal Rank Fusion (RRF), which combines the two ranked lists by (a small constant ) and needs no score calibration between the incomparable BM25 and cosine scales.
Common mistakes
- Comparing BM25 scores across queries. BM25 is a within-query ranking score, not a calibrated probability — a score of 12 for one query is not "more relevant" than 8 for another. Only the order within one query is meaningful.
- Skipping tokenization/normalization. BM25 matches surface tokens, so lowercasing, stemming/lemmatization, and stop-word handling directly change recall. Mismatched analyzers between indexing and querying silently tank results.
- Leaving / at defaults for atypical corpora. Very short docs (tweets, titles) or very long ones (manuals) often want different ; tune on labeled queries.
- Expecting semantic matches. BM25 will never connect "heart attack" to "myocardial infarction" — that gap is exactly what the dense half of a hybrid retriever is for.
- Feeding negative IDF into the sum. Use the -inside-log form (above) so ultra-common terms don't subtract from a document's score.
Related concepts
- Retrieval-Augmented Generation: BM25 is the classic first-stage retriever, and the sparse half of hybrid RAG
- Embeddings & Semantic Search: the dense counterpart BM25 is fused with in hybrid search
- Vector Databases & ANN Indexes: the dense-retrieval index BM25's inverted index complements