BM25: a score built from rare words, counts and length
BM25 is a retriever that gives a passage a weight for each word of the query. The sum of these weights is the passage's score. Each weight comes from three things: how rare the word is in the whole corpus, how many times it appears in the passage, and how long the passage is. Stanford's slide calls it the most widely used retriever, even today.
The need for it showed up in lesson 2. counts how many query words also appear in a passage. For the query which office hour is least crowded it gave two passages the same score of 2. One is the queue log, which holds the crowd data and the answer. The other is an old forum post from 2024, which holds only an opinion. This lesson adds BM25's three parts one at a time and checks the tie each time.
The ladder for this lesson
The ladder has 5 steps. Under each step you can see what you will be able to do after it.
What you need first: Lesson 2 (corpus, word overlap, stopwords, score), Lesson 3 (vector, embedding) and Lesson 4 (e and ln). A word with a dotted underline and ▾ opens its definition when you click it.
1IDF: more weight for rare terms
IDF gives every word a weight, and the fewer passages a word appears in, the bigger its weight. In search, each such word of the query is called a term. are very common words such as which and is, which are removed before counting. After removing them, our query has four terms left: office, hour, least and crowded.
Office appears in two passages, and least and crowded only in the forum post, yet word overlap counts all four the same. The number of passages a term appears in is called its document frequency, written n. Our corpus has N = 6 passages, and milestone appears only in the syllabus, so n = 1. Its IDF takes three steps:
Count the passages without the term and add 0.5: 6 − 1 + 0.5 = 5.5.
Count the passages with the term and add 0.5 as well: 1 + 0.5 = 1.5.
Divide the first number by the second, 5.5 ÷ 1.5 = 3.667, and take its ln: ln 3.667 = 1.299.
The ratio in step 3 says how many passages lack the term for each passage that has it. The 0.5 is added so that the ratio still exists when a count is 0.
The in step 3 comes from lesson 4. ln x is the power you raise e = 2.718 to in order to get x. Check: e1.2993 = 3.667, so ln 3.667 = 1.299. One rule will be useful below: ln 1 = 0, because e0 = 1.
This weight is called IDF, short for inverse document frequency: the smaller n is, the bigger the IDF. The formula holds the same three steps:
IDF = ln N − n + 0.5n + 0.5
Term, and the passages it is in
n
Arithmetic
IDF
milestonesyllabus
1
ln(5.5 ÷ 1.5)
1.299
officeoffice hours page, queue log
2
ln(4.5 ÷ 2.5)
0.588
thursdaysyllabus, office hours page, queue log
3
ln(3.5 ÷ 3.5)
0
Thursday is in half the passages, so its ratio is 1, and since ln 1 = 0, its weight is 0.
Now take the terms of our query. Office has an IDF of 0.588. Hour appears only in the queue log, because the office hours page says hours, not hour, and the retriever treats these as different words. So hour, least and crowded all have n = 1 and an IDF of 1.299.
Think first
For now, count each term only once and add up its IDF. What do the forum post and the queue log score, and which way does the tie break?
Show the answer
With the IDFs to 4 decimals, the forum post gets 1.2993 + 1.2993 = 2.599, and the queue log 0.5878 + 1.2993 = 1.887. The tie breaks, but towards the forum post, because both of its terms are rare, while the queue log's office is common.
But office hour appears twice in the queue log, for Tuesday and for Thursday, and IDF counts it only once. The next step adds this count.
2Term frequency and saturation: a count that pays off only up to a ceiling
The term frequency f is how many times a term appears in one passage. In the queue log, office and hour both have f = 2. But the first appearance tells you the passage is about that term, and the tenth adds very little to that. So BM25 does not simply multiply by f.
It turns f into a new number, which this lesson calls the TF part. Leave passage length aside for now, it comes in step 3:
TF part = f × 2.2f + 1.2
For f = 2 the arithmetic takes three steps:
Top: 2 × 2.2 = 4.4.
Bottom: 2 + 1.2 = 3.2.
Divide: 4.4 ÷ 3.2 = 1.375.
Check: at f = 1 the top is 2.2 and the bottom is 2.2, so the TF part is exactly 1. Now let f grow:
f
TF part
1
1.000
2
1.375
5
1.774
10
1.964
Going from 1 to 2 appearances adds 0.375, while going from 5 to 10 adds only 0.190. As f grows, the 1.2 in the bottom becomes small next to it, so the TF part gets close to 2.2 but never reaches it. This is called saturation: each extra appearance adds less than the one before, and the TF part never goes above a ceiling.
The 1.2 in the formula is a knob called k1, and the ceiling 2.2 is really k1 + 1. A small k1 means a low ceiling that fills up quickly. k1 = 1.2 is my choice, and the slide's graph has this line too
Now back to the queue log. Office and hour both appear twice, so both have a TF part of 1.375, and the score is 0.588 × 1.375 + 1.299 × 1.375 = 2.595. Both of the forum post's terms appear once, so its score stays at 2.599. Adding the count brings the queue log almost level, with a gap of only 0.004.
Think first
What does the TF part become when k1 = 0? How does that relate to the scores in step 1?
Show the answer
The formula becomes f × 1 ÷ (f + 0), that is, f ÷ f = 1, as long as f is at least 1. That is the word overlap of lesson 2, only with IDF weights. The 2.599 and 1.887 from step 1 were exactly these k1 = 0 scores.
Stanford's slide: the TF part for four values of k1 on the right, and IDF on the left.
3Length normalization: less weight for a term in a long passage
Length normalization lowers the TF part of a term in a long passage, and raises it for a term in a short passage. Office appearing once in a 1,000-word document says less than office appearing once in a 20-word passage.
For this, BM25 compares each passage's length with the average length in the corpus. A passage's length in words is written |D|, and the average length is avgdl. An average is the sum of all the lengths divided by the number of passages:
Check: 16.17 × 6 = 97.02, which is about 97. Now BM25 multiplies the k1 in the bottom of the TF part by this factor:
factor = 1 − b + b × |D|avgdl
|D| ÷ avgdl says how many times the average a passage is. b is a second knob, between 0 and 1, that sets how much length matters. For the queue log, with b = 0.75, the factor takes three steps: b = 0.75 is my choice
|D| ÷ avgdl = 19 ÷ 16.17 = 1.175.
Multiply by b: 0.75 × 1.175 = 0.881.
Add 1 − b: 0.25 + 0.881 = 1.131.
Check: at average length the factor is 0.25 + 0.75 = 1, so nothing changes. A longer passage has a factor above 1, so its TF part shrinks. At f = 1, for example, a passage twice the average length drops from 1.000 to 0.710.
This is called length normalization. With b = 0 the factor is always 1, so length has no effect, and with b = 1 length has its full effect. In our corpus the queue log has 19 words, more than average, so its factor is 1.131. The forum post has 14 words, so its factor is 0.25 + 0.75 × 14 ÷ 16.17 = 0.899.
Think first
Say b = 1 and the passage is twice the average length. What is the TF part at f = 1 and k1 = 1.2?
Show the answer
Factor = 1 − 1 + 1 × 2 = 2. TF part = 2.2 ÷ (1 + 1.2 × 2) = 2.2 ÷ 3.4 = 0.647. That is even less than the 0.710 at b = 0.75, because length now has its full effect.
4All three parts in one formula: BM25
The BM25 score multiplies each query term's IDF by its TF part, and adds this up over all the terms. A term that is not in the passage contributes 0. Do the whole calculation for office in the queue log: f = 2, and the factor from step 3, 1.1314 to 4 decimals.
Bottom: 2 + 1.2 × 1.1314 = 2 + 1.3577 = 3.3577.
Top: 2 × 2.2 = 4.4.
TF part: 4.4 ÷ 3.3577 = 1.3104.
Multiply by the IDF: 0.5878 × 1.3104 = 0.7703.
For least in the forum post the factor is 0.8995, so the TF part is 2.2 ÷ (1 + 1.2 × 0.8995) = 2.2 ÷ 2.0794 = 1.0580. For the query which office hour is least crowded, with k1 = 1.2 and b = 0.75, the full calculation is:
BM25 ranks the old forum post above the queue log, 2.749 against 2.473. Length widened the gap of 0.004 from step 2 to 0.276. The formula only counts words, and no part of it can tell that the forum post is a 2024 opinion and the queue log is fresh data.
Stanford's slide writes the same calculation as one formula:
score(D, Q) = ∑i = 1 … n IDF(qi) · f(qi, D) · (k1 + 1)f(qi, D) + k1 · (1 − b + b · |D| ÷ avgdl)
The comes from lesson 3. It means: for every i from the number written below up to the n written above, work out what follows, and add it all up. Here i runs from 1 to 4: 0.7703 + 1.7026 + 0 + 0 = 2.473. The “·” in the formula is a multiplication sign, D is the passage being scored, and qi is the i-th query term. The other parts come from steps 1 to 3.
Stanford's slide: BM25, the most widely used. The coloured parts are IDF, f and |D| ÷ avgdl, and below is the IDF formula from step 1.
Work out the office hours page's row
The office hours page has 16 words, and office appears in it once. What are its factor, TF part and score?
Show the answer
16 ÷ 16.17 = 0.9897, so the factor is 0.25 + 0.75 × 0.9897 = 0.9923. TF part = 2.2 ÷ (1 + 1.2 × 0.9923) = 2.2 ÷ 2.1908 = 1.0042. Score = 0.5878 × 1.0042 = 0.590.
BM25 calculator
Pick a query and change k1 and b. The three buttons are the first three steps, and each cell shows the term's f and its weight.
Passage
|D|
Terms
Score
On the third query, when can I get help from a TA, every score is 0, and the reason comes in step 5.
5The inverted index: why BM25 is fast, and where it fails
An inverted index is a list, for each term, of the passages that contain it. BM25 builds these lists , that is, once, before any question arrives. The lists for our query's terms look like this:
office → office hours page, queue log
hour → queue log
least → old forum post
crowded → old forum post
When a question arrives, BM25 opens only these 4 lists and never touches the other 3 passages. That is why it is fast: on Stanford's comparison slide it answers in 62 ms on a large dataset.
To BM25, every passage is a long vector with one slot for each word in the vocabulary. The forum post has 14 of our 69 words, so 55 of its slots are 0, and a vector like this is called sparse. The of lesson 3 are dense: 768 numbers, almost none of them 0.
Stanford's slide: Fast vs Slow Retrievers, its BM25 row. The other rows come in lessons 12, 14 and 15.
According to the same row, BM25 wins on exact terms, such as IDs, names and error codes. A code like E-4012 is either in a passage or not, and because it is rare, its IDF is high.
BM25 breaks on paraphrase and synonyms, that is, the same thing said in other words. The answer to the widget's third query is on the office hours page, but that page says TAs, not ta, and help appears nowhere. So every score is 0.
The forum post mistake comes from the same place, because BM25 only counts words. Lesson 12 builds a retriever that learns from question-passage pairs, and lesson 14 combines both lists.
What the dense version adds
6. The slide's symbols, the two meanings of n, k1 + 1, and the full length table.
7. Two more ln rules, negative IDF, TF-IDF, where the 0.5 comes from, and both graphs.
8. Who picks k1 and b, and at which values the queue log comes out on top.
9. What the index stores, the sparse dot product, the 62 ms data, and examples from the papers.
10. All four Stanford slides, which you read yourself.
BM25 score = each query term's IDF × its TF part, summed, with k1 making the count saturate and b accounting for length.
The inverted index makes BM25 fast, and it is reliable on exact terms.
BM25 counts words and cannot tell data from opinion. So the old forum post, at 2.749, ranks above the queue log (2.473).
New terms in this lesson
Term
One word of the query in search. For example, office and hour in which office hour.
Document frequency, n(q)
How many documents contain term q. For example, n = 1 for milestone and 2 for office.
IDF
ln((N − n + 0.5) ÷ (n + 0.5)), a bigger weight for a rarer term. For example, 1.299 for milestone and 0.588 for office.
TF-IDF
The term's count × ln(N ÷ n), an older method than BM25. For example, N = 6, n = 2, f = 2 gives 2 × ln 3 = 2.20.
Term frequency, f(q, D)
How many times term q appears in document D. For example, office has f = 2 in the queue log.
Saturation and k1
As the count grows, the gain fills up slowly towards a ceiling of k1 + 1. For example, with k1 = 1.2 the TF part for f = 1, 2, 10 is 1.000, 1.375, 1.964.
Length normalization, b and avgdl
Lowering the TF part of documents longer than average, by an amount set by b. For example, with avgdl = 16.17 and b = 0.75, a document twice the average length gets 0.710.
BM25
Each query term's IDF × its TF part, summed over the terms. For example, 2.749 for the forum post and 2.473 for the queue log.
Inverted index
For each term, the list of documents that contain it. For example, office → office hours page, queue log.
Sparse and dense vector
A long vector that is mostly zeros, and a short one that is almost full. For example, 55 of the 69 slots in the forum post's vector are 0.