Lesson 14 · Easy version

Hybrid search and reranking

Hybrid search merges the ranked lists of two retrievers into one list. Reranking then sorts the top passages of that list again, using a slower but more accurate model.

BM25 put E2 at rank 3, and dense retrieval put it at rank 1. The query was E-4012 upload failed, and it has two correct passages. E1 contains this error code exactly, and E2 says the same thing in other words. Lesson 12 ended on this point: the strengths and weaknesses of are the opposite of a dense retriever's.

The ladder for this lesson

The ladder has 5 steps. Under each step you can see what you will be able to do after it.

  1. BM25 and dense retrieval miss different passagesSay what kind of text each retriever misses.
  2. Hybrid search, and simply adding the scoresShow why adding the scores buries the dense retriever's order.
  3. Reciprocal rank fusion: ranks instead of scoresCompute RRF by hand, and explain what k = 60 does.
  4. Cross-encoder: query and passage togetherDo one attention step by hand, and explain why a cross-encoder is slow.
  5. Reranking a shortlistWork out the time reranking takes, and explain why a reranker cannot raise recall.

What you need first: Lesson 3 (embedding, cosine), Lesson 6 (fusion, rerank), Lesson 11 (BM25), Lesson 12 (bi-encoder, DPR) and Lesson 13 (MRR). If you already know a step, skip to the next one. A word with a dotted underline and ▾ opens its definition when you click it.

1BM25 and dense retrieval miss different passages

BM25 and a dense retriever fail on different kinds of text, so on the same query they leave out different correct passages. BM25 counts the words a query and a passage share, while a dense retriever such as compares their .

BM25 fails when a passage says the same thing in other words, which is called a paraphrase. For example, the query says upload failed and the passage says upload keeps failing. To BM25, failing and failed are different words, so the only match it finds is upload.

A dense retriever fails when something has to match exactly, such as an error code or an order ID. Text like this is called a literal. Lesson 3 showed that passages with E-4012 and E-4021 can get almost the same embedding. To BM25, 4012 is a rare word with a high IDF, so it finds the right passage.

Now look at how both retrievers rank three passages: my example

PassageCorrect?BM25Dense
E1, the one with the error codeError E-4012: file larger than 50 MB. Compress the file and upload again.YesRank 14012 matches exactlyRank 2
E2, the one in other wordsIf your upload keeps failing on Gradescope, make the file smaller before submitting.YesRank 3only upload matchesRank 1closest vector
E3, the one with another errorError E-4021: submission locked after the deadline.NoRank 2an E- code, short passageRank 3

At top-1, BM25 brings only E1 and dense retrieval brings only E2. Each misses a different correct passage, so merging their lists can bring both correct passages to the top.

Real data shows the same thing. In lesson 12, top-20 accuracy on SQuAD was 68.8% for BM25 and 63.2% for DPR. On the same Stanford slide, a retriever that combines the two reaches 71.5%, ahead of both.

Think first

The query is refund policy for order #A77-310. Which part will BM25 catch, and which part will the dense retriever catch?

Show the answer

BM25 catches the rare literal A77-310 exactly. The dense retriever can match refund policy to a passage about money back rules, which shares no word with it.

2Hybrid search, and simply adding the scores

Hybrid search runs both retrievers and merges their two ranked lists into one list. Lesson 6 called merging lists . The simplest way is to add each passage's two scores: the BM25 score and the dense retriever's . my scores

Think first

The BM25 scores of E1, E3 and E2 are 7.2, 3.1 and 1.0, and their cosines are 0.79, 0.75 and 0.82. If you add the two, where does E2, the dense retriever's first passage, end up?

Show the answer

At rank 3, the bottom. The table below has the arithmetic.

PassageBM25 scoreCosinePlain sum
E17.20.797.99
E3 (wrong)3.10.753.85
E21.00.821.82

After adding, the order is E1, E3, E2, exactly BM25's order. The reason is the scale of the two scores. The cosines spread from 0.75 to 0.82, only 0.07, while BM25 spreads from 1.0 to 7.2, which is 6.2.

BM25's spread is about 90 times the cosine's, so in the sum the cosine has almost no effect. A cosine always stays between −1 and 1, while BM25 has no upper limit. You could divide BM25 by some number, but that number would differ for every corpus, and you would have to find it each time.

3Reciprocal rank fusion: merging ranks instead of scores

The scores have different scales, but ranks have the same scale in every retriever: first, second, third. Reciprocal rank fusion, or RRF, turns a passage's rank in each list into 1 ÷ (60 + rank), and adds these numbers over all lists. The passages are then ordered by this sum.

E1 is at rank 1 in BM25's list, so it gets 1 ÷ (60 + 1) = 1 ÷ 61 = 0.01639, about 0.0164. In the dense list it is at rank 2, so it gets 1 ÷ 62 = 0.01613. E1's RRF is the sum: 0.01639 + 0.01613 = 0.03252.

PassageFrom BM25From denseSum
E11 ÷ (60 + 1) = 0.016391 ÷ (60 + 2) = 0.016130.03252
E21 ÷ (60 + 3) = 0.015871 ÷ (60 + 1) = 0.016390.03227
E3 (wrong)1 ÷ (60 + 2) = 0.016131 ÷ (60 + 3) = 0.015870.03200

The new order is E1, E2, E3. Both correct passages moved to the top, and E3 dropped to the bottom, because it was not first in any list.

The three sums sit close together, and the reason is how a fraction behaves. When the bottom of 1 ÷ (60 + rank) grows, the fraction shrinks. But going from 61 to 62 makes the bottom only 1.6% bigger, so the fraction also shrinks by only about 1.6%. That is why RRF numbers stay close together, and only their order matters.

Stanford's slide writes the same arithmetic as a formula:

RRF(d) = ∑r ∈ R 1k + rankr(d),   k = 60
d
A passage, such as E1.
R
The set of retrievers, here {BM25, dense}. The r ∈ R under the means: add over every retriever r in R.
rankr(d)
The rank of d in retriever r's list. For E1, 1 in BM25 and 2 in dense. A list that does not contain d adds nothing.
k
A fixed number, 60. It is not the k of top-k.
Stanford slide: Hybrid Search and Reciprocal Rank Fusion
Stanford's slide: Hybrid Search and Reciprocal Rank Fusion. The first line is step 1, the second is step 2, and at the bottom is the RRF formula. The small line says RRF needs no per-corpus tuning and works with any number of retrievers. Click the image to open it at full size.

What k = 60 does

k = 60 shrinks the gap between rank 1 and rank 2, so rank 1 in a single list cannot decide the result on its own. With k = 0, rank 1 would get 1 and rank 2 would get 0.5, half as much. With k = 60, they get 0.01639 and 0.01613.

Suppose passage A is only in BM25's list, at rank 1, and passage B is at rank 3 in both lists: my example

So with k = 60, the winner is the passage that both retrievers place near the top.

Change k yourself

A is at rank 1 in one list, and B is at rank r in both lists. Change k and r and see how long B stays ahead.

60 3
PassageArithmeticRRF

Think first

In lesson 6, the original query's list was: queue log, forum post, office hours page. The rewritten query's list was: office hours page, queue log. What order does RRF give?

Show the answer

Queue log: 1 ÷ 61 + 1 ÷ 62 = 0.03252. Office hours page: 1 ÷ 63 + 1 ÷ 61 = 0.03227. The forum post is only in the first list: 1 ÷ 62 = 0.01613. The order is queue log, office hours page, forum post.

The price of RRF is that it forgets how strong a score was. Whether E1's BM25 score is 7.2 or 70, rank 1 gives 1 ÷ 61 both times.

4Cross-encoder: reading the query and the passage together

E1 and E2 are at the top of the RRF list only because the two retrievers agreed on them. Neither retriever read the query and the passage together. In a , a passage's vector is made before any question arrives, and the two meet only at the final dot product.

A cross-encoder joins the query and the passage into one text, feeds it to one model, and gets a single score for that pair. The model is usually an encoder such as .

Attention: each token updates its vector from the other tokens

An encoder makes a vector for every and then updates it many times. In attention, each token takes a weighted average of the other tokens' vectors and uses it as its new vector. In a weighted average you multiply each vector by its weight and add them up, and the weights add up to 1.

In a cross-encoder, the query's tokens can also look at the passage's tokens, which Stanford's slide calls full attention across both. Below, every vector has 2 numbers: the first says how much the token leans toward counted data, and the second how much it leans toward opinion. my numbers

The query token crowded starts as [0.5, 0.5]. When the query is joined with the queue log, crowded gives weight 0.6 to the queue log's averaged [1.0, 0.0] and 0.4 to itself:

  1. First number: 0.6 × 1.0 + 0.4 × 0.5 = 0.6 + 0.2 = 0.8.
  2. Second number: 0.6 × 0.0 + 0.4 × 0.5 = 0 + 0.2 = 0.2.

With the forum post, the same 0.6 goes to usually [0.0, 1.0], and crowded becomes [0.2, 0.8]. So the same query token gets a different vector with every passage. In a bi-encoder, attention runs only inside the query, so crowded keeps one vector for every passage.

At the end, the model gathers the whole pair into one vector and turns it into a score. The reranker in lesson 6 gave the queue log 0.92, the office hours page 0.41 and the forum post 0.35 on the least-crowded question. The retriever had given the queue log and the forum post the same 2. my scores from lesson 6

Stanford slide: Cross-Encoders and Reranking, diagrams of a bi-encoder and a cross-encoder
Stanford's slide: Cross-Encoders and Reranking. On the left, in the bi-encoder, the query (Inx) and the passage (Iny) go through separate encoders. On the right, in the cross-encoder, both go through one Encoder together.

The price: nothing can be built in advance

Every vector in a cross-encoder depends on the query, so nothing about a passage can be computed before the question arrives. On every query it has to run the whole model once for every passage, while a bi-encoder ran the model once per query.

According to Stanford's slide, a cross-encoder built on BERT-base takes 10,700 ms for 1,000 passages, which is 10.7 ms per passage. BERT-base is the smaller version of BERT. For the 60,000 passages of the company in lesson 2, one query would take 60,000 × 10.7 ms = 642 seconds, about 11 minutes.

5Reranking a shortlist

A retriever is fast but looks at passages roughly, and a cross-encoder is accurate but slow. So a fast retriever first picks the top few passages, such as the top 100, and then the cross-encoder scores and sorts only those again. This short list is called the shortlist, and the second step is reranking.

StepOn whatTime
1. BM25 and dense, then RRFThe whole corpusBM25 62 msdense tens of ms, from Stanford's slide
2. Reranking with the cross-encoderTop-100 shortlist100 × 10.7 = 1,070 ms
3. Into the LLM's promptTop 5 after reranking–

The whole corpus would take 642 seconds, and the shortlist takes about 1 second. A Stanford table shows the gain in accuracy. Its data is MS MARCO: real Bing search questions and about 8.8 million passages. MS MARCO description from the paper

There, BM25 alone gets an of 16.7. In lesson 13 this was one over the rank of the first correct passage, such as 0.5 at rank 2. The table writes it multiplied by 100. Rerank BM25's top 1,000 with a BERT-base cross-encoder, and MRR@10 becomes 34.7. So reranking roughly doubles MRR@10: 34.7 ÷ 16.7 = 2.08. BM25's top 1,000 from the paper

A reranker has a limit: it only changes the order of the shortlist, and it cannot bring in a passage that is not there. This is the from lesson 2. If the shortlist holds the right passage for 60 out of 100 questions, the reranker cannot go above 60 either.

So the first step needs high recall, and hybrid search provides it: both E1 and E2 reach the shortlist.

Think first

Each query has a budget of 500 ms. The retriever takes 50 ms, and the cross-encoder takes 10.7 ms per passage. At most how many passages can the shortlist hold?

Show the answer

That leaves 500 − 50 = 450 ms for reranking. 450 ÷ 10.7 = 42.06, so 42 passages.

What the dense version adds

The dense version contains this whole lesson, followed by 5 steps at the Stanford level:

  • Step 6: the maths of RRF: 1 ÷ (k + rank) for different k, and the RRF paper's experiment on k.
  • Step 7: min-max normalisation, its weakness, and the DPR paper's BM25 + DPR combination.
  • Step 8: the cross-encoder from inside: attention weights, score to probability, and the training loss.
  • Step 9: the cost in full: running the model N times against once, FLOPs, and the real recall ceiling.
  • Step 10: five Stanford slides, which you read yourself.

Open the dense version

3 points from this lesson

  1. BM25 misses paraphrases, and a dense retriever misses rare literals. Hybrid search runs both and merges their lists.
  2. The scores have different scales, so RRF merges only ranks: 1 ÷ (60 + rank) from each list, then a sum. With k = 60, the passage that both lists place high wins.
  3. A cross-encoder reads the query and the passage together, so it is accurate, but it runs in full for every passage. That is why it reranks only a shortlist.

New terms in this lesson

Paraphrase
The same statement in other words. For example, upload failed and upload keeps failing. BM25 misses it, because it counts shared words.
Literal
Text that has to match exactly, such as the error code E-4012 or the order ID A77-310. A dense retriever can miss it, and BM25 catches it as a rare word.
Hybrid search
Running both a word-based and a dense retriever and merging their lists. For example, making one list from BM25's E1, E3, E2 and dense retrieval's E2, E1, E3.
Reciprocal rank fusion (RRF)
1 ÷ (k + rank) from a passage's rank in each list, summed over all lists, with k = 60. For example, E1: 1 ÷ 61 + 1 ÷ 62 = 0.03252.
Attention
Each token takes a weighted average of the other tokens' vectors as its new vector, and the weights add up to 1. For example, if crowded [0.5, 0.5] gives weight 0.6 to averaged [1.0, 0.0], it becomes [0.8, 0.2].
Cross-encoder
A reranker that joins the query and the passage, feeds them to one model, and gets a direct score for the pair. It runs in full for every passage, for example 10.7 ms per passage with BERT-base.
Reranking
Scoring and sorting a fast retriever's shortlist again with an accurate model. For example, on MS MARCO, MRR@10 goes from 16.7 for BM25 alone to 34.7 after reranking with BERT-base.
Shortlist
The retriever's top few passages, on which the reranker runs. For example, the top 100 of 60,000 passages, which the cross-encoder handles in 1,070 ms.