BM25: a score built from rare words, counts and length
BM25 is a retriever that gives a passage a weight for each word of the query. The sum of these weights is the passage's score. Each weight comes from three things: how rare the word is in the whole corpus, how many times it appears in the passage, and how long the passage is. Stanford's slide calls it the most widely used retriever, even today.
The need for it showed up in lesson 2. counts how many query words also appear in a passage. For the query which office hour is least crowded it gave two passages the same score of 2. One is the queue log, which holds the crowd data and the answer. The other is an old forum post from 2024, which holds only an opinion. This lesson adds BM25's three parts one at a time and checks the tie each time.
The ladder for this lesson
The ladder has 10 steps. The first 5 are the same as in the easy version, and the Stanford level starts at step 6. Under each step you can see what you will be able to do after it.
What you need first: Lesson 2 (corpus, word overlap, stopwords, score), Lesson 3 (vector, dot product, embedding) and Lesson 4 (e and ln). If you already know a step, skip to the next one. If you get stuck, go back one step. A word with a dotted underline and ▾ opens its definition when you click it.
1IDF: more weight for rare terms
IDF gives every word a weight, and the fewer passages a word appears in, the bigger its weight. In search, each such word of the query is called a term. are very common words such as which and is, which are removed before counting. After removing them, our query has four terms left: office, hour, least and crowded.
Office appears in two passages, and least and crowded only in the forum post, yet word overlap counts all four the same. The number of passages a term appears in is called its document frequency, written n. Our corpus has N = 6 passages, and milestone appears only in the syllabus, so n = 1. Its IDF takes three steps:
Count the passages without the term and add 0.5: 6 − 1 + 0.5 = 5.5.
Count the passages with the term and add 0.5 as well: 1 + 0.5 = 1.5.
Divide the first number by the second, 5.5 ÷ 1.5 = 3.667, and take its ln: ln 3.667 = 1.299.
The ratio in step 3 says how many passages lack the term for each passage that has it. The 0.5 is added so that the ratio still exists when a count is 0.
The in step 3 comes from lesson 4. ln x is the power you raise e = 2.718 to in order to get x. Check: e1.2993 = 3.667, so ln 3.667 = 1.299. One rule will be useful below: ln 1 = 0, because e0 = 1.
This weight is called IDF, short for inverse document frequency: the smaller n is, the bigger the IDF. The formula holds the same three steps:
IDF = ln N − n + 0.5n + 0.5
Term, and the passages it is in
n
Arithmetic
IDF
milestonesyllabus
1
ln(5.5 ÷ 1.5)
1.299
officeoffice hours page, queue log
2
ln(4.5 ÷ 2.5)
0.588
thursdaysyllabus, office hours page, queue log
3
ln(3.5 ÷ 3.5)
0
Thursday is in half the passages, so its ratio is 1, and since ln 1 = 0, its weight is 0.
Now take the terms of our query. Office has an IDF of 0.588. Hour appears only in the queue log, because the office hours page says hours, not hour, and the retriever treats these as different words. So hour, least and crowded all have n = 1 and an IDF of 1.299.
Think first
For now, count each term only once and add up its IDF. What do the forum post and the queue log score, and which way does the tie break?
Show the answer
With the IDFs to 4 decimals, the forum post gets 1.2993 + 1.2993 = 2.599, and the queue log 0.5878 + 1.2993 = 1.887. The tie breaks, but towards the forum post, because both of its terms are rare, while the queue log's office is common.
But office hour appears twice in the queue log, for Tuesday and for Thursday, and IDF counts it only once. The next step adds this count.
2Term frequency and saturation: a count that pays off only up to a ceiling
The term frequency f is how many times a term appears in one passage. In the queue log, office and hour both have f = 2. But the first appearance tells you the passage is about that term, and the tenth adds very little to that. So BM25 does not simply multiply by f.
It turns f into a new number, which this lesson calls the TF part. Leave passage length aside for now, it comes in step 3:
TF part = f × 2.2f + 1.2
For f = 2 the arithmetic takes three steps:
Top: 2 × 2.2 = 4.4.
Bottom: 2 + 1.2 = 3.2.
Divide: 4.4 ÷ 3.2 = 1.375.
Check: at f = 1 the top is 2.2 and the bottom is 2.2, so the TF part is exactly 1. Now let f grow:
f
TF part
1
1.000
2
1.375
5
1.774
10
1.964
Going from 1 to 2 appearances adds 0.375, while going from 5 to 10 adds only 0.190. As f grows, the 1.2 in the bottom becomes small next to it, so the TF part gets close to 2.2 but never reaches it. This is called saturation: each extra appearance adds less than the one before, and the TF part never goes above a ceiling.
The 1.2 in the formula is a knob called k1, and the ceiling 2.2 is really k1 + 1. A small k1 means a low ceiling that fills up quickly. k1 = 1.2 is my choice, and the slide's graph has this line too
Now back to the queue log. Office and hour both appear twice, so both have a TF part of 1.375, and the score is 0.588 × 1.375 + 1.299 × 1.375 = 2.595. Both of the forum post's terms appear once, so its score stays at 2.599. Adding the count brings the queue log almost level, with a gap of only 0.004.
Think first
What does the TF part become when k1 = 0? How does that relate to the scores in step 1?
Show the answer
The formula becomes f × 1 ÷ (f + 0), that is, f ÷ f = 1, as long as f is at least 1. That is the word overlap of lesson 2, only with IDF weights. The 2.599 and 1.887 from step 1 were exactly these k1 = 0 scores.
Stanford's slide: the TF part for four values of k1 on the right, and IDF on the left.
3Length normalization: less weight for a term in a long passage
Length normalization lowers the TF part of a term in a long passage, and raises it for a term in a short passage. Office appearing once in a 1,000-word document says less than office appearing once in a 20-word passage.
For this, BM25 compares each passage's length with the average length in the corpus. A passage's length in words is written |D|, and the average length is avgdl. An average is the sum of all the lengths divided by the number of passages:
Check: 16.17 × 6 = 97.02, which is about 97. Now BM25 multiplies the k1 in the bottom of the TF part by this factor:
factor = 1 − b + b × |D|avgdl
|D| ÷ avgdl says how many times the average a passage is. b is a second knob, between 0 and 1, that sets how much length matters. For the queue log, with b = 0.75, the factor takes three steps: b = 0.75 is my choice
|D| ÷ avgdl = 19 ÷ 16.17 = 1.175.
Multiply by b: 0.75 × 1.175 = 0.881.
Add 1 − b: 0.25 + 0.881 = 1.131.
Check: at average length the factor is 0.25 + 0.75 = 1, so nothing changes. A longer passage has a factor above 1, so its TF part shrinks. At f = 1, for example, a passage twice the average length drops from 1.000 to 0.710.
This is called length normalization. With b = 0 the factor is always 1, so length has no effect, and with b = 1 length has its full effect. In our corpus the queue log has 19 words, more than average, so its factor is 1.131. The forum post has 14 words, so its factor is 0.25 + 0.75 × 14 ÷ 16.17 = 0.899.
Think first
Say b = 1 and the passage is twice the average length. What is the TF part at f = 1 and k1 = 1.2?
Show the answer
Factor = 1 − 1 + 1 × 2 = 2. TF part = 2.2 ÷ (1 + 1.2 × 2) = 2.2 ÷ 3.4 = 0.647. That is even less than the 0.710 at b = 0.75, because length now has its full effect.
4All three parts in one formula: BM25
The BM25 score multiplies each query term's IDF by its TF part, and adds this up over all the terms. A term that is not in the passage contributes 0. Do the whole calculation for office in the queue log: f = 2, and the factor from step 3, 1.1314 to 4 decimals.
Bottom: 2 + 1.2 × 1.1314 = 2 + 1.3577 = 3.3577.
Top: 2 × 2.2 = 4.4.
TF part: 4.4 ÷ 3.3577 = 1.3104.
Multiply by the IDF: 0.5878 × 1.3104 = 0.7703.
For least in the forum post the factor is 0.8995, so the TF part is 2.2 ÷ (1 + 1.2 × 0.8995) = 2.2 ÷ 2.0794 = 1.0580. For the query which office hour is least crowded, with k1 = 1.2 and b = 0.75, the full calculation is:
BM25 ranks the old forum post above the queue log, 2.749 against 2.473. Length widened the gap of 0.004 from step 2 to 0.276. The formula only counts words, and no part of it can tell that the forum post is a 2024 opinion and the queue log is fresh data.
Stanford's slide writes the same calculation as one formula:
score(D, Q) = ∑i = 1 … n IDF(qi) · f(qi, D) · (k1 + 1)f(qi, D) + k1 · (1 − b + b · |D| ÷ avgdl)
The comes from lesson 3. It means: for every i from the number written below up to the n written above, work out what follows, and add it all up. Here i runs from 1 to 4: 0.7703 + 1.7026 + 0 + 0 = 2.473. The “·” in the formula is a multiplication sign, D is the passage being scored, and qi is the i-th query term. The other parts come from steps 1 to 3.
Stanford's slide: BM25, the most widely used. The coloured parts are IDF, f and |D| ÷ avgdl, and below is the IDF formula from step 1.
Work out the office hours page's row
The office hours page has 16 words, and office appears in it once. What are its factor, TF part and score?
Show the answer
16 ÷ 16.17 = 0.9897, so the factor is 0.25 + 0.75 × 0.9897 = 0.9923. TF part = 2.2 ÷ (1 + 1.2 × 0.9923) = 2.2 ÷ 2.1908 = 1.0042. Score = 0.5878 × 1.0042 = 0.590.
BM25 calculator
Pick a query and change k1 and b. The three buttons are the first three steps, and each cell shows the term's f and its weight.
Passage
|D|
Terms
Score
On the third query, when can I get help from a TA, every score is 0, and the reason comes in step 5.
5The inverted index: why BM25 is fast, and where it fails
An inverted index is a list, for each term, of the passages that contain it. BM25 builds these lists , that is, once, before any question arrives. The lists for our query's terms look like this:
office → office hours page, queue log
hour → queue log
least → old forum post
crowded → old forum post
When a question arrives, BM25 opens only these 4 lists and never touches the other 3 passages. That is why it is fast: on Stanford's comparison slide it answers in 62 ms on a large dataset.
To BM25, every passage is a long vector with one slot for each word in the vocabulary. The forum post has 14 of our 69 words, so 55 of its slots are 0, and a vector like this is called sparse. The of lesson 3 are dense: 768 numbers, almost none of them 0.
Stanford's slide: Fast vs Slow Retrievers, its BM25 row. The other rows come in lessons 12, 14 and 15.
According to the same row, BM25 wins on exact terms, such as IDs, names and error codes. A code like E-4012 is either in a passage or not, and because it is rare, its IDF is high.
BM25 breaks on paraphrase and synonyms, that is, the same thing said in other words. The answer to the widget's third query is on the office hours page, but that page says TAs, not ta, and help appears nowhere. So every score is 0.
The forum post mistake comes from the same place, because BM25 only counts words. Lesson 12 builds a retriever that learns from question-passage pairs, and lesson 14 combines both lists.
From here: the Stanford level
The steps above are the same as in the easy version. The steps below take you to the level of the Stanford slides: symbols, graphs, k1 and b, real scale, and all four slides.
6Every symbol of the slide's formula
The slide writes BM25 as two formulas: one for the whole score, and one for IDF. You read the score formula in step 4. This step ties every symbol of both formulas to a number from the queue log.
score(D, Q) = ∑i = 1 … n IDF(qi) · f(qi, D) · (k1 + 1)f(qi, D) + k1 · (1 − b + b · |D| ÷ avgdl)
IDF(qi) = ln N − n(qi) + 0.5n(qi) + 0.5
Some parts of the slide carry coloured labels. The table below includes those labels too:
Symbol
Meaning
In our example
Q, D
The query, and the passage being scored
Our query, and D = the queue log
∑, i = 1 … n
Sum over all n query terms
n = 4
qi
The i-th query term
q1 = office, q2 = hour, q3 = least, q4 = crowded
IDF(qi)
The term's weight, label How rare qi is
0.588 for office
f(qi, D)
How many times the term is in the passage, label How often qi occurs in D
office 2, hour 2, least 0, crowded 0
k1
The saturation knob
1.2
b
The knob for how much length matters
0.75
|D| ÷ avgdl
Passage length against the average, label Doc length vs. the average
19 ÷ 16.17 = 1.175
N
Total documents in the corpus, label Docs in the collection
6
n(qi)
How many documents contain the term, label Docs that contain qi
2 for office
0.5
A small number added to each count
4.5 ÷ 2.5 for office
score(D, Q)
The sum of all the term weights
0.7703 + 1.7026 + 0 + 0 = 2.473
Note that n appears twice on the slide. The n above the ∑ is the number of query terms, and the n(qi) in the IDF formula is the document frequency. They are different things.
What the k1 + 1 on top does
The k1 + 1 in the numerator multiplies every term's TF part by the same number, 2.2. So every passage's whole score is multiplied by 2.2 as well, and the order of the passages does not change. If you drop k1 + 1, for example, the forum post comes to 2.749 ÷ 2.2 = 1.250 and the queue log to 2.473 ÷ 2.2 = 1.124. The forum post is still on top.
According to Robertson and Zaragoza, many versions keep k1 + 1 so that a term that appears once in an average-length passage gets exactly its IDF as its weight. from the paper With our numbers: at f = 1 and factor = 1 the TF part is 2.2 ÷ 2.2 = 1, so the weight is IDF × 1.
The effect of length: the full table
Step 3 gave the TF part for two lengths. Here is the full table for three lengths, at f = 1, k1 = 1.2 and b = 0.75:
Passage length
Factor
TF part
Half the average
0.25 + 0.75 × 0.5 = 0.625
1.257
The average
0.25 + 0.75 × 1 = 1
1.000
Twice the average
0.25 + 0.75 × 2 = 1.75
0.710
The same single appearance earns 1.257 in a short passage and 0.710 in a long one. At twice the length, for example: 2.2 ÷ (1 + 1.2 × 1.75) = 2.2 ÷ 3.1 = 0.710. For the queue log, |D| ÷ avgdl = 1.175, so it is 17.5% longer than average.
Shrinking the count by length
There is a second way to read the length factor. It needs one rule about ratios: dividing the top and the bottom by the same number does not change a ratio. For example, 6 ÷ 3 = 2, and (6 ÷ 2) ÷ (3 ÷ 2) = 3 ÷ 1.5 = 2.
Divide the top and the bottom of the TF part by the factor. The k1 × factor in the bottom becomes just k1, and f is replaced by f ÷ factor:
So BM25 first shrinks the count by length, and then applies the saturation from step 2. Robertson and Zaragoza write BM25 exactly this way. from the paper For office in the queue log it takes three steps:
The shrunk count: 2 ÷ 1.1314 = 1.7677.
The formula from step 2: top 1.7677 × 2.2 = 3.8889, bottom 1.7677 + 1.2 = 2.9677.
Divide: 3.8889 ÷ 2.9677 = 1.3104, the same TF part as in step 4.
In the forum post, the single count of least becomes 1 ÷ 0.8995 = 1.1117, and 1.1117 × 2.2 ÷ (1.1117 + 1.2) = 2.4457 ÷ 2.3117 = 1.0580. Length turned the queue log's 2 counts into something like 1.77, and the forum post's 1 count into something like 1.11.
7The edges of IDF, and both graphs on the slide
The left graph on the slide shows BM25's IDF and the older IDF, and the right graph shows the TF part for four values of k1. To read them, first look at the edges of IDF: very common and very rare terms. Exact numbers are hard to read off a graph, so this step works out the same numbers from the formulas. my arithmetic
The edges of IDF: two more ln rules, negative IDF and TF-IDF
Step 1 gave one rule of ln: ln 1 = 0. Reading the slide's IDF graph needs two more:
The ln of a number below 1 is negative, and flipping a ratio only flips the sign of its ln. For example, ln 2 = 0.693, so ln(1 ÷ 2) = −0.693, because e−0.693 = 0.5.
The ln of a product is the sum of the lns: ln(a × b) = ln a + ln b. For example, ln 4 = ln(2 × 2) = 0.693 + 0.693 = 1.386.
The second rule means that when a ratio grows 10 times, its ln grows by only ln 10 = 2.303. So ln squeezes very large ratios into small weights. A term that appears in just 1 of 10,000 passages has a ratio of 9,999.5 ÷ 1.5 = 6,666, but an IDF of only 8.805.
The first rule produces negative IDF. A term that appears in more than half the passages has a ratio below 1. A term in 5 of the 6 passages, for example, has the ratio 1.5 ÷ 5.5, which is milestone's ratio 5.5 ÷ 1.5 flipped, so its IDF is −1.299. At the other end, for a term that appears in no passage, the ratio without the 0.5 would be 6 ÷ 0, which has no value.
The older TF-IDF, which came before BM25, uses the weight ln(N ÷ n). N ÷ n says the term appears in one passage out of how many. Office is in 2 of 6 passages, that is, 1 in every 3, so its weight is ln 3 = 1.099. This weight is never negative, and TF-IDF multiplies it directly by the count: for office in the queue log, 2 × 1.099 = 2.20.
The left graph: BM25's IDF and the older IDF
The left graph uses N = 10,000 documents. The blue line is BM25's IDF, and the green dashed line is the standard IDF, that is, the ln(N ÷ n) of TF-IDF. The graph's source code uses exactly these two formulas. from the figure's source The x-axis shows the share of documents that contain the term. For example, 10% means 1,000 of the 10,000 documents.
Documents with the term
BM25 IDF
ln(N ÷ n)
0.01%n = 1
8.805
9.210
1%n = 100
4.590
4.605
10%n = 1,000
2.197
2.303
50%n = 5,000
0
0.693
90%n = 9,000
−2.197
0.105
100%n = 10,000
−9.904
0
The arithmetic for the 10% row: for BM25, (10,000 − 1,000 + 0.5) ÷ (1,000 + 0.5) = 9,000.5 ÷ 1,000.5 = 8.996, and ln 8.996 = 2.197. For the standard IDF, 10,000 ÷ 1,000 = 10, and ln 10 = 2.303. The table shows these points:
For rare terms the two are almost equal, which is why the two lines run together on the left of the graph.
At 50%, BM25's IDF is exactly 0, because the top and the bottom are both 5,000.5, and ln 1 = 0. The graph's red dashed line is this 50% threshold.
Past 50%, BM25's IDF is negative, and that is the graph's Negative IDF region. The 90% ratio is the 10% ratio flipped, so by the flipped-ratio rule the IDF only changes sign: from +2.197 to −2.197.
The standard IDF never goes below 0, because n is never larger than N, and at 100% it is ln 1 = 0.
A negative IDF means that finding such a term lowers a passage's score. According to the figure's source, many implementations therefore clamp negative IDF at 0, or drop such terms before scoring. from the figure's source In our corpus the stopword the is in 3 passages, exactly half, so its IDF is 0.
Where the 0.5 comes from: the RSJ weight worked out in full
BM25's IDF comes from an older weight by Robertson and Spärck Jones, called the RSJ weight. from the paper The RSJ weight is built from four counts. N and n are the same as before. R is the number of documents that are really useful for the query, called relevant. r is how many of those relevant documents contain the term.
w = ln (r + 0.5) · (N − R − n + r + 0.5)(n − r + 0.5) · (R − r + 0.5)
First, suppose we know that the syllabus is relevant to a question about the milestone. Then R = 1 and r = 1, with N = 6 and n = 1:
In the second calculation the 0.5 on top and the 0.5 below cancel, and ln((N − n + 0.5) ÷ (n + 0.5)) is left. Without the 0.5s this calculation would be (0 × 5) ÷ (1 × 0) = 0 ÷ 0, which has no value. That is why the paper adds 0.5 to every count, so that the ratio and the log never break. from the paper
The right graph: four values of k1, four curves
The right graph shows the TF part from step 2 for four values of k1, without length, that is, at factor = 1. Each curve's ceiling is k1 + 1. Some points from the formula:
k1
f = 2
f = 10
f = 20
0.5ceiling 1.5
1.200
1.429
1.463
1.2ceiling 2.2
1.375
1.964
2.075
2.0ceiling 3
1.500
2.500
2.727
5.0ceiling 6
1.714
4.000
4.800
The arithmetic for the cell at k1 = 5 and f = 20: top 20 × 6 = 120, bottom 20 + 5 = 25, and 120 ÷ 25 = 4.8. In the graph every curve starts at 0 for f = 0, passes through 1 at f = 1, and then bends towards its ceiling. The k1 = 0.5 curve flattens early, and at f = 20 it is very close to its ceiling of 1.5. The k1 = 5 curve is still climbing at f = 20, because 4.8 is well below its ceiling of 6.
At f = 1 every curve is at 1, because the top and the bottom are both 1 + k1. No curve ever reaches its ceiling: at k1 = 1.2, even f = 100 gives 220 ÷ 101.2 = 2.174. Also look at how much each extra appearance adds: the first adds 1.000, the second 0.375, the third 1.571 − 1.375 = 0.196, and the tenth 1.964 − 1.941 = 0.023.
According to the paper, this formula did not come straight out of a deeper model. BM25's authors saw a saturating shape in a probability model of term counts, and fitted the simplest formula with that shape, f ÷ (k + f). from the paper
8Who picks k1 and b
k1 and b are hyperparameters: BM25 does not learn them from data, the person building the search picks them. A hyperparameter is a number in a formula or model that training does not change. BM25 has no training at all, so k1 and b are the only numbers in it that anyone chooses.
The usual method is to set aside some questions whose correct passages are known. Then try many values of k1 and b, see how high the correct passages come out, and keep the best pair. Such a set of questions is called a dev set, short for development set. Lesson 13 teaches how to measure whether the correct passages came out on top.
According to Robertson and Zaragoza, many experiments find that b from 0.5 to 0.8 and k1 from 1.2 to 2 work well. They also say the best values depend on the kind of documents and queries. from the paper
The paper also explains why b sits between 0 and 1. A document can be long for two reasons. One, the author says the same thing in more words, and then dividing the count fully by length is right, that is, b = 1. Two, the document really covers more topics, and then dividing by length is wrong, that is, b = 0. Real documents mix both reasons. from the paper
The DPR paper ran Lucene's BM25 on Wikipedia passages and picked k1 = 0.9 and b = 0.4 on its dev sets. Both values lie outside the range above. from the paper Even with these values the order on our query stays the same: forum post 2.666, queue log 2.420. my arithmetic
Think first
In the widget from step 4, set b = 0 and move k1 from 1.2 to 1.3. What happens to the forum post and the queue log, and why?
Show the answer
The queue log, at 2.630, moves above the forum post (2.599). With b = 0 length has no effect, and a larger k1 gives more weight to the queue log's two-times counts. With b = 0 the order flips at k1 ≈ 1.21.
With b = 0.75 the flip needs k1 of about 3.13. The widget's slider stops at 3, where the forum post is still ahead, 2.810 against 2.799.
This flip does not happen because of the queue log's data. The queue log writes office hour twice, and a larger k1 gave more weight to repetition. The person building the search picks k1 and b once for the whole corpus, not separately for each question. A setting that lifts the queue log on this question can lift, on other questions, passages that repeat a word over and over.
Think first
Your company's help center has a Complete user guide that is 10 times as long as the average article. Every common term appears in it about 20 times, and it comes out on top for almost every query. Which way should k1 and b move?
Show the answer
Raise b and lower k1. With k1 = 1.2 and b = 0.75 the guide's factor is 0.25 + 0.75 × 10 = 7.75, and its TF part is 20 × 2.2 ÷ (20 + 1.2 × 7.75) = 44 ÷ 29.3 = 1.502. An average-length article with the term once has a TF part of 1.000.
With b = 1 the guide's factor is 10, and its TF part is 44 ÷ (20 + 12) = 1.375. Setting k1 = 0.5 as well gives 20 × 1.5 ÷ (20 + 0.5 × 10) = 30 ÷ 25 = 1.200. The article stays at 1.000, so the gap shrinks from 0.502 to 0.200. Then measure on a dev set whether the right articles come out on top.
9BM25 at real scale
In a real inverted index, every entry in a term's list stores the passage together with its f. The lists for our query's terms look like this:
office → (office hours page, f = 1), (queue log, f = 2)
hour → (queue log, f = 2)
least → (old forum post, f = 1)
crowded → (old forum post, f = 1)
The length of a list is n. Office's list has 2 entries, for example, so n = 2, and the IDF follows from that. The index also stores each passage's |D| and the corpus's avgdl. When a question arrives, BM25 reads 5 entries across 4 lists and scores only 3 passages.
BM25 as a dot product of sparse vectors
According to the DPR paper, TF-IDF and BM25 treat the question and the passage as very long sparse vectors, with a weight for each word. from the paper To see this you need the from lesson 3. A dot product multiplies the numbers in the same position of two vectors and adds them up. For example, the dot product of [1, 0, 1] and [2, 5, 3] is 1 × 2 + 0 × 5 + 1 × 3 = 5.
For our corpus, make a vector with 69 numbers, one for each word in the vocabulary. In a passage's vector, each word's slot holds its weight, IDF × TF part, and the slot of a word not in the passage holds 0. In the query's vector, the slots of the 4 query terms hold 1, and the other 65 slots hold 0.
The dot product of these two vectors is exactly the BM25 score, because a product is non-zero only where both vectors have a number. For the forum post those are the two slots for least and crowded: 1 × 1.3747 + 1 × 1.3747 = 2.749. In the other 67 slots, at least one side is 0. An inverted index is really a way of storing these sparse vectors by term, so that the zero slots never have to be read.
What data the 62 ms was measured on
Below Stanford's comparison slide it says the latencies come from Khattab and Zaharia (2020), the ColBERT paper, on MS MARCO. In that paper, 62 ms is the number for Anserini's BM25, and Anserini is a search toolkit built on Lucene. from the paper
MS MARCO holds 8.8 million passages from web pages, gathered from Bing's results for 1 million real questions. In the paper each query returned the top 1,000 of these 8.8 million passages, and latency was measured on one CPU thread. According to the paper, BM25 is this fast because its steps are cheap and this kind of search has been optimized for decades. from the paper
In the DPR paper, Lucene's BM25 handles 23.7 questions per second on one CPU thread. Since 1 second = 1,000 ms, that is 1,000 ÷ 23.7 = 42 ms per question. Its inverted index over 21 million passages takes about 30 minutes to build, while DPR's vectors and index take 8.8 + 8.5 = 17.3 hours. from the paper The two papers use different hardware and corpora, so 62 ms and 42 ms cannot be compared directly, only that both are in the tens of milliseconds.
BM25's strength and weakness in the papers
The DPR paper says methods like BM25 are sensitive to highly selective keywords and phrases. from the paper The IDs, names and error codes in the slide's row are words of this kind. An example from the same paper shows the weakness. The question is: Who is the bad guy in lord of the rings? Its answer is in this passage: Sala Baker is best known for portraying the villain Sauron in the Lord of the Rings trilogy.
The question says bad guy, and the passage says villain. So, according to the paper, a term-based system would have difficulty finding this passage, while a dense retriever can bring the vectors for bad guy and villain close together. from the paper This is a real example of the slide's Breaks on: Paraphrase, synonyms.
10Now read Stanford's slides yourself
Below are the four BM25 slides, exactly as they are. Under each slide are a few questions. Think of the answer first, then open it. If a symbol stops you, go back to the step named in the answer.
If a slide looks small on a phone, click the image. It opens at full size.
Stanford's slide: Different Types of Retrievers.
Read the slide on types of retrievers
The slide calls BM25 word-overlap. What three things does BM25 add to the word overlap of lesson 2?
Show the answer
More weight for rare terms (IDF), a count that pays off only up to a ceiling (k1), and a lower TF part for long passages (b). With k1 = 0, BM25 is just word overlap with IDF weights. (Steps 1, 2 and 3)
What is the score built from in word-overlap and in vector retrieval?
Show the answer
In BM25 the score comes only from terms that appear exactly in both the query and the passage, a dot product of long sparse vectors. In vector retrieval an encoder turns each text into a short dense vector, and the score is the dot product of those vectors. (Steps 5 and 9)
Stanford's slide: BM25, the most widely used, and below it both formulas, enlarged.
Read the BM25 slide
What is the IDF(qi) in the yellow box for hour? What are the numbers in the green and pink boxes?
Show the answer
Green is N = 6, and pink is n(hour) = 1, because hour appears only in the queue log. IDF = ln((6 − 1 + 0.5) ÷ (1 + 0.5)) = ln(5.5 ÷ 1.5) = 1.299. (Steps 1 and 6)
What is the blue box's |D| ÷ avgdl for the queue log, and what does the whole bracket around it come to?
Show the answer
19 ÷ 16.17 = 1.175. The bracket is 1 − 0.75 + 0.75 × 1.175 = 1.131, the factor from step 3. (Step 3)
If you drop the (k1 + 1) on top, does the order of the passages change?
Show the answer
No. Every term's part is divided by 2.2, so every score is divided by 2.2. The forum post comes to 1.250 and the queue log to 1.124, and the order stays the same. (Step 6)
Stanford's slide: IDF Comparison and Term Frequency Saturation, and below it both graphs, enlarged.
Read the slide with the graphs
Why does the blue line in the left graph cross 0 exactly at 50%?
Show the answer
At 50%, n = N ÷ 2, so the top, N − n + 0.5, equals the bottom, n + 0.5. The ratio is 1, and ln 1 = 0. (Steps 1 and 7)
Why does the green dashed line never go below 0, and where are the two lines at 100%?
Show the answer
The green line is ln(N ÷ n), and n is never larger than N, so the ratio is at least 1. At 100% it sits at ln 1 = 0. The blue line there falls, by the formula, to ln(0.5 ÷ 10,000.5) = −9.904. (Step 7)
Why do all four lines in the right graph pass through the same point at f = 1? And what height is the k1 = 5 line heading for?
Show the answer
At f = 1 the TF part is (k1 + 1) ÷ (1 + k1) = 1, whatever k1 is. The k1 = 5 line is heading for its ceiling of 6, and at f = 20 it is at 4.8. The graph has no length in it, so it shows the factor = 1 case. (Steps 2 and 7)
Stanford's slide: Fast vs Slow Retrievers, and below it the BM25 row, enlarged. The other rows come in lessons 12, 14 and 15.
Read the comparison slide
The Precomputed offline column says Inverted index for BM25. In our corpus, what is already built before a query arrives?
Show the answer
A list for each term, where each entry holds a passage and its f. The length of a list gives n, and n gives the IDF. Also each passage's |D| and avgdl = 16.17. (Steps 5 and 9)
Under what conditions was the 62 ms measured, and how fast was BM25 in the DPR paper?
Show the answer
Anserini's BM25, the 8.8 million passages of MS MARCO, the top 1,000 per query, and one CPU thread. In the DPR paper, Lucene's BM25 handles 23.7 questions per second, about 42 ms per question. The slide itself notes that the DPR numbers come from different hardware. (Step 9)
Show the Wins on and Breaks on cells with our corpus.
Show the answer
If a rare code like E-4012 is in a single passage, its IDF is high and that passage comes out on top. On when can I get help from a TA every score is 0, because the page with the answer says TAs, and help is not there at all. (Steps 1 and 5)
3 points from this lesson
BM25 score = each query term's IDF × its TF part, summed over the terms. IDF gives weight to rare terms, k1 makes the count saturate, and b shrinks the count by length.
k1 and b are hyperparameters, picked once for the whole corpus on a dev set. On our query the usual values keep the forum post on top. With b = 0 and k1 = 1.3 the queue log comes out on top, but only because of repetition.
Because of the inverted index, BM25 reads only the lists of the query's terms, so it runs in tens of milliseconds and is reliable on exact terms. It fails on queries in different words, and it does not see whether a passage holds data or an opinion.
New terms in this lesson
Term
One word of the query in search. For example, office and hour in which office hour.
Document frequency, n(q)
How many documents contain term q. For example, n = 1 for milestone and 2 for office.
IDF
ln((N − n + 0.5) ÷ (n + 0.5)), a bigger weight for a rarer term. For example, 1.299 for milestone and 0.588 for office.
TF-IDF
The term's count × ln(N ÷ n), an older method than BM25. For example, N = 6, n = 2, f = 2 gives 2 × ln 3 = 2.20.
Term frequency, f(q, D)
How many times term q appears in document D. For example, office has f = 2 in the queue log.
Saturation and k1
As the count grows, the gain fills up slowly towards a ceiling of k1 + 1. For example, with k1 = 1.2 the TF part for f = 1, 2, 10 is 1.000, 1.375, 1.964.
Length normalization, b and avgdl
Lowering the TF part of documents longer than average, by an amount set by b. For example, with avgdl = 16.17 and b = 0.75, a document twice the average length gets 0.710.
BM25
Each query term's IDF × its TF part, summed over the terms. For example, 2.749 for the forum post and 2.473 for the queue log.
Hyperparameter
A number in a formula or model that training does not change, picked by trying values on a dev set. For example, BM25's k1 and b, which the DPR paper set to 0.9 and 0.4.
Inverted index
For each term, the list of documents that contain it. For example, office → office hours page, queue log.
Sparse and dense vector
A long vector that is mostly zeros, and a short one that is almost full. For example, 55 of the 69 slots in the forum post's vector are 0.