A vector is a list of numbers with a fixed length. A retriever turns every text into a vector, then scores each passage by comparing the query's vector with the passage's vector. There are two common ways to make this comparison, the dot product and cosine similarity, and in this lesson you compute both by hand.
The need for this showed up at the end of lesson 2. For the query when can I get help from a TA, gave every passage 0. The answer was on the office hours page: Office hours… run by the TAs. The two say the same thing, but they share not a single word.
The ladder for this lesson
The ladder has 9 steps. The first 4 are the same as in the easy version, and steps 5 to 9 take you to the level of the Stanford slides. Under each step you can see what you will be able to do after it.
What you need first: Lesson 2 (query, passage, score, top-K, word overlap). From step 5 on, a few terms come from later lessons, such as distance from lesson 10 and BM25 from lesson 11. If you already know a step, skip to the next one. A word with a dotted underline and ▾ opens its definition when you click it.
1A list of numbers for a text
A vector is a list of numbers with a fixed length, such as [0.9, 0.1]. How many numbers the list holds is called the vector's dimension. [0.9, 0.1] has dimension 2, and [3, 0, 1, 2] has dimension 4. A retriever needs one score per passage, and numbers are easy to compare, so it turns every text into a vector.
A vector with 2 numbers can be drawn on a plane as an arrow: the first number goes on the x-axis and the second on the y-axis. The arrow from the origin (0, 0) to that point is the vector, and the arrow's direction is the vector's direction.
Now rate each passage by hand on two questions, giving each answer a value between 0 and 2. The first number says how much the passage is about help or office hours. The second number says how much it is about deadlines. Take three passages from lesson 2's corpus. Then add a new one: the course's long FAQ page, which says a lot about both office hours and deadlines. my vectors
Text
Vector
Office hours page
[0.9, 0.1]
Syllabus: the milestone date
[0.1, 0.9]
Project page: the proposal date
[0.2, 0.8]
Course FAQ, a long page
[1.6, 1.4]
Query: when can I get help from a TA
[0.8, 0.2]
The query and the office hours page share no word, yet their vectors are close together: [0.8, 0.2] and [0.9, 0.1]. These numbers are the answers to the two questions, and the two texts got almost the same answers. So a vector can capture a match that counting words misses.
In real systems the dimensions have no such names, and there are hundreds of them, for example 768 in BERT-base vectors. from the paper Here there are 2, so that you can see the vectors on a plane. Every calculation below works the same way with 768 numbers.
Think first
Rate the passage Extra office hours on Zoom this Friday on the same two questions. Roughly what vector do you get, and which passage in the table is it close to?
Show the answer
This passage is entirely about office hours and mentions no deadline. So you get a vector of about [0.9, 0.0], which sits right next to the office hours page at [0.9, 0.1].
2Dot product: the long vector wins
The dot product turns two vectors into one number: multiply the numbers in the same position, then add up all the products. For the query [0.8, 0.2] and the office hours page [0.9, 0.1], it is 0.8 × 0.9 + 0.2 × 0.1 = 0.72 + 0.02 = 0.74. Another name for the dot product is the inner product, and Stanford's slides use that name.
Think first
The FAQ page [1.6, 1.4] does not point in the query's direction as closely as the office hours page does. Will its dot product be smaller or larger than 0.74?
Show the answer
Larger: 0.8 × 1.6 + 0.2 × 1.4 = 1.28 + 0.28 = 1.56. The table below has the other passages too.
Passage
Arithmetic
Dot product
Office hours page[0.9, 0.1]
0.8 × 0.9 + 0.2 × 0.1 = 0.72 + 0.02
0.74
Syllabus[0.1, 0.9]
0.8 × 0.1 + 0.2 × 0.9 = 0.08 + 0.18
0.26
Project page[0.2, 0.8]
0.8 × 0.2 + 0.2 × 0.8 = 0.16 + 0.16
0.32
FAQ page[1.6, 1.4]
0.8 × 1.6 + 0.2 × 1.4 = 1.28 + 0.28
1.56
The FAQ page comes out on top because both of its numbers are large. Its vector is exactly twice [0.8, 0.7]. The query's dot product with [0.8, 0.7] is 0.64 + 0.14 = 0.78, and with the FAQ page it is 1.56 = 2 × 0.78.
Doubling a vector always does this. Every multiplication in a dot product takes one number from that vector. When that number doubles, every product doubles, and so does their sum. So the dot product measures the arrow's length along with its direction, and a long vector can come out on top just because it is long.
In symbols, the same arithmetic is written like this:
a · b = ∑i = 1 … daibi
∑ (sigma) means: keep adding for every i in the range written below it, here from i = 1 up to the dimension d = 2. ai is the query's i-th number, and bi is the office hours page's. So a · b = a1b1 + a2b2 = 0.8 × 0.9 + 0.2 × 0.1 = 0.74.
The dot product is fast, only d multiplications and d additions, and the retrievers in lesson 5 and lesson 12 use it as their score. Its limit showed up above: a passage with a long vector can come out on top even when its direction matches less well.
3Cosine similarity: comparing direction only
Cosine similarity compares only the direction of two vectors, not their length. To do this, it divides the dot product by the product of the two vectors' lengths. So first, learn to find a length.
A vector's length
A vector's length is the length of its arrow. On the plane, the query's arrow is the longest side of a right triangle whose other two sides are 0.8 and 0.2 long. So by Pythagoras, the length is √(0.8² + 0.2²). In other words, square each number, add the squares, then take the square root:
Squares: 0.8 × 0.8 = 0.64, and 0.2 × 0.2 = 0.04.
Sum: 0.64 + 0.04 = 0.68.
Square root: √0.68 = 0.82. The square root is the number that gives the inside number when you multiply it by itself. Check: 0.8246 × 0.8246 = 0.680.
|a| = √( ∑iai2 )
In the same way, the office hours page has length √(0.81 + 0.01) = √0.82 = 0.91.
The dot product divided by both lengths
Now divide the dot product by the product of the two lengths. For the query and the office hours page that is 0.74 ÷ (0.82 × 0.91) = 0.99. This number is called the cosine similarity.
cos(a, b) = a · b|a| × |b|
It is called cosine because this number is the cosine of the angle between the two arrows. For arrows pointing the same way it is 1, at 90° it is 0, and for opposite directions it is −1. The query and the office hours page are about 8° apart, and cos 8° = 0.99.
Here is the same query, with length 0.82, against every passage:
Passage
Dot product
Length
Cosine
Office hours page[0.9, 0.1]
0.74
0.91
0.990.74 ÷ (0.82 × 0.91)
FAQ page[1.6, 1.4]
1.56
2.13
0.891.56 ÷ (0.82 × 2.13)
Project page[0.2, 0.8]
0.32
0.82
0.470.32 ÷ (0.82 × 0.82)
Syllabus[0.1, 0.9]
0.26
?
?
The lengths are rounded to 2 decimals. Each cosine comes from the unrounded numbers, so the last digit of a hand calculation can differ slightly.
Work out the syllabus row
Syllabus = [0.1, 0.9], and its dot product with the query is 0.26. What are the syllabus's length and cosine?
Show the answer
Length = √(0.01 + 0.81) = √0.82 = 0.91. Cosine = 0.26 ÷ (0.82 × 0.91) = 0.35. It is at the bottom, because the syllabus is about deadlines and the query is about getting help.
With cosine, the office hours page comes first and the FAQ page drops to second place. The FAQ page's vector is twice [0.8, 0.7], so its length is twice as large too: 2.126 instead of 1.063. Its dot product had also doubled, and in the division the 2 cancels. So a vector's length has no effect on cosine, and the long page loses its advantage.
Normalizing: vectors of length 1
There is also a shortcut. Divide every number in a vector by the vector's length, and its length becomes 1 while its direction stays the same. This is called normalizing the vector. The normalized query is [0.8 ÷ 0.8246, 0.2 ÷ 0.8246] = [0.970, 0.243], and the normalized office hours page is [0.994, 0.110].
The dot product of normalized vectors is the cosine itself, because both lengths in the denominator are now 1: 0.970 × 0.994 + 0.243 × 0.110 = 0.964 + 0.027 = 0.991. So if every vector in the index is normalized, the dot product and cosine give the same score.
Drag the query's arrow
Click or drag on the plane, and the query's arrow moves there. For the milestone query, the dot product puts the FAQ page on top, and cosine puts the syllabus on top.
Passage
Vector
Dot product
Cosine
Cosine has a limit too: it ignores length completely, and sometimes length carries useful information. A way of turning two vectors into a similarity number is called a similarity metric. Which metric to use depends on the metric the model that makes the vectors was trained with.
4Embedding model: a vector for any text, automatically
The vectors above were made by hand, by asking two questions. That cannot be done by hand for the 60,000 passages of lesson 2's company, and two questions cannot capture everything a text says. So we need a program that gives any text a vector automatically.
An embedding is the vector a trained model produces for a text, so that texts saying similar things get vectors pointing in similar directions. The model that reads a text and produces its embedding is called an embedding model or an encoder. An encoder is also made of the from lesson 1.
Training shows the encoder pairs of texts that say the same thing, and pairs that do not. It then changes the parameters so that the score of the first kind of pair goes up and the score of the second kind goes down. Lesson 12 covers this training in full.
Now return to the TA query. Use the encoder to make the query's embedding, and compute its cosine with lesson 2's passages. The result might look like this: my numbers, not from a real model
Passage
Shared words
Cosine
Office hours pageOffice hours: Tuesday 4pm in Room 104 and Thursday 9am on Zoom, run by the TAs.
Homework 1 announcementHomework 1 will be released on Oct 5.
0
0.22
SyllabusGrading. The project milestone is due Week 6 …
0
0.15
The query and the office hours page share no word, yet the office hours page is on top with 0.78. Where lesson 2's retriever gave every passage 0, the embedding retriever brings the right passage to the top.
Offline and online
An embedding retriever runs in two parts. Offline, before any query arrives, the encoder makes each passage's embedding once. This list of embeddings, built in advance, is called the index. Online, on every query, the encoder makes only the query's embedding. Then its cosine or dot product with every embedding in the index is computed, and the K passages with the highest scores come back.
Stanford's DPR slide shows these same two parts. The DPR of lesson 12 has two separate encoders: EP for passages and EQ for the query. The EQ(q)⊤ EP(p) in the slide's box is the dot product of the two vectors, and the is one way of writing it.
Stanford's slide: the offline part of DPR at the top, and the online part at the bottom. Click the image to open it at full size.
An embedding retriever has two limits that later lessons deal with. First, it can miss things that must match exactly: passages with error codes E-4012 and E-4021 can get almost the same embedding. Lesson 14 solves this. Second, every query is compared with every embedding in the index, so 60,000 passages mean 60,000 cosines per query. With tens of millions of passages this gets slow, and lesson 10 solves that.
Think first
A new passage is added to the corpus. What does the embedding retriever have to do?
Show the answer
The encoder makes the embedding of that one passage, and you add it to the index. The rest of the index stays as it is. Each query still needs only one new embedding, the query's.
From here: the Stanford level
The 4 steps above are the same as in the easy version. The 5 steps below take you to the level of the Stanford slides: first the maths of cosine and angles, then how the dot product, cosine and distance relate and why DPR chose one of them. After that comes the index as a matrix, then the five kinds of embeddings in Stanford's table, and finally you read the slides yourself. Every piece of maths is explained where it first appears.
5Cosine and the angle: one number for direction
Cosine similarity really is the cosine of the angle between the two arrows, which is why it always stays between −1 and 1. Step 3 only stated this. This step shows it on our numbers, using two rules from school.
The numbers of an arrow of length 1
The first school rule is about right triangles. The cos of an angle is the side next to it divided by the longest side, and the sin is the side opposite it divided by the longest side. Now take an arrow of length 1 that makes an angle A with the x-axis. Its longest side is 1, so its two numbers are simply cos A and sin A.
In step 3 the normalized query was [0.970, 0.243], an arrow of length 1. Its first number is cos A, and cos 14.0° = 0.970, so the query's arrow is at 14.0° from the x-axis. In the same way, the office hours page [0.994, 0.110] is at 6.3°. The angle between the two arrows is 14.0° − 6.3° = 7.7°. my arithmetic
The rule for cos(A − B)
The second school rule is about the difference of two angles:
cos(A − B) = cos A · cos B + sin A · sin B
The right-hand side is exactly the dot product of two arrows of length 1: multiply the first numbers, multiply the second numbers, then add. On our numbers:
Query: cos A = 0.970 and sin A = 0.243. Office hours page: cos B = 0.994 and sin B = 0.110.
Cosine of the angle: cos 7.7° = 0.991. Check: the two are equal.
So the dot product of normalized vectors is the cosine of the angle between them, and in step 3 this dot product was the cosine similarity. The table below shows the angles of all four passages, and every row gives the same cos as the table in step 3:
Passage
Angle from x-axis
Angle from query
cos
Office hours page
6.3°
7.7°
0.99
FAQ page
41.2°
27.1°
0.89
Project page
76.0°
61.9°
0.47
Syllabus
83.7°
69.6°
0.35
The query is at 14.0° from the x-axis. The angles come from the unrounded numbers, so subtracting the rounded angles can change the last digit. For the FAQ page, 41.19° − 14.04° = 27.15°, and cos 27.15° = 0.89.
Check the rule on three easy cases yourself. The dot product of [1, 0] and [1, 1] is 1, and their lengths are 1 and √2 = 1.414, so the cosine is 1 ÷ 1.414 = 0.707. On the plane the arrow [1, 1] sits at exactly 45°, and cos 45° = 0.707. The dot product of [1, 0] and [0, 1] is 0, so the angle is 90°. The cosine of [0.8, 0.2] and [−0.8, −0.2] is −0.68 ÷ 0.68 = −1, so the angle is 180°.
A dot product is never larger than the product of the two lengths
The cos of any angle lies between −1 and 1. So cosine = a · b ÷ (|a| × |b|) stays in that range too, which means a · b is never larger than |a| × |b|. Look at the FAQ page: its dot product is 1.56, and 0.82 × 2.13 = 1.75, so 1.56 is smaller. This rule also holds in 768 dimensions, where you cannot draw the arrows, and there the same formula is what defines the angle.
The two are equal only when both arrows point the same way. For example, [0.4, 0.1] is exactly half of the query: the dot product is 0.32 + 0.02 = 0.34, and 0.8246 × 0.4123 = 0.34 as well, so the cosine is 1.00. By dot product this short vector would rank below the office hours page (0.74), but by cosine it comes first. my example
Think first
All the numbers in our vectors are 0 or larger. Can the cosine of any pair then be below 0? What happens with real embeddings?
Show the answer
It cannot. If every number in both vectors is 0 or larger, every product is 0 or larger, so the dot product and the cosine are too. On the plane all the arrows lie in the first quadrant, and the angle between two of them cannot exceed 90°. Real embeddings also contain negative numbers, so there the cosine can drop below 0.
6Dot product, cosine and distance: when they agree and when they don't
Distance, the dot product and cosine are tied together by one formula: on normalized vectors all three give the same order. On other vectors their orders can differ. The between two vectors is the length of their difference. Office hours page − query = [0.9 − 0.8, 0.1 − 0.2] = [0.1, −0.1], and its length is √(0.01 + 0.01) = √0.02 = 0.14. The smaller the distance, the closer the passage.
Here is the order of our four passages under all three: my arithmetic
Passage
Dot product
Cosine
Distance
Office hours page
0.742nd
0.991st
0.141st
FAQ page
1.561st
0.892nd
1.444th
Project page
0.323rd
0.473rd
0.852nd
Syllabus
0.264th
0.354th
0.993rd
The FAQ page is first by dot product, second by cosine, and last by distance. One formula links the three, and it needs one school rule: (x − y)² = x² + y² − 2xy. For example, (0.9 − 0.8)² = 0.81 + 0.64 − 1.44 = 0.01.
Expanding the squared distance
The squared distance is the sum of the squares of the difference vector's numbers. Apply the rule above to each number, then add everything up. The x² parts add up to |a|², the y² parts to |b|², and the xy parts to a · b:
|a − b|2 = |a|2 + |b|2 − 2 a · b
Office hours page: 0.68 + 0.82 − 2 × 0.74 = 1.50 − 1.48 = 0.02, and √0.02 = 0.14. Check: the same 0.14 as above.
The FAQ page's |b|² = 4.52 is very large, so even its large dot product cannot make the distance small. In other words, the dot product rewards length, distance penalizes it, and cosine ignores it.
Now normalize both vectors. Then |a|² = |b|² = 1, and the formula becomes 2 − 2 cos. For the office hours page that is 2 − 2 × 0.991 = 0.018, and for the FAQ page 2 − 2 × 0.890 = 0.220. The larger the cosine, the smaller the distance, so all three give the same order. So on normalized vectors, the choice among the three does not change the ranking.
The DPR paper says the same: for unit vectors, meaning vectors of length 1, cosine and the inner product are equivalent. It adds that most similarity functions of this kind are transformations of L2 distance. from the paper L2 distance is just this distance's name.
Why DPR chose the dot product
The DPR paper sets one condition on the similarity function: it must be decomposable. That means the score comes from a question vector and a passage vector, and the passage vector is built without seeing the question. Only then can step 4's offline index exist. The dot product, cosine and distance all meet this condition. A model that reads the question and the passage together does not, and the of lesson 14 is such a model. from the paper
The paper tried cosine and L2 distance alongside the dot product. In one place it says the other functions performed comparably. In the experiments section it states that L2 was comparable to the dot product and that both were better than cosine. On the NQ dev set, top-20 accuracy is 78.1% with the dot product and 76.1% with L2. The paper gives no numbers for cosine. from the paper So DPR kept the simplest one, the dot product.
Other models are trained with cosine. Sentence-BERT compares its vectors with cosine, and E5's score is cosine ÷ 0.01. from the paper
In E5, a pair with cosine 0.99 scores 99, and one with 0.89 scores 89. Every score is divided by the same number, so the order stays the same, just as it did when a vector was doubled in step 2. The rule is to retrieve with the same metric the model was trained with.
7The index is a matrix: every score in one multiplication
Stanford's slide says retrieval needs a fixed-dimension vector per chunk. The dimension has to be fixed because the dot product pairs up numbers by position. [0.8, 0.2] and [0.9, 0.1, 0.5] have no dot product, because the third number has no partner. That is why the FAQ page, long as it is, still has only 2 numbers in its vector, and every passage vector in DPR has 768.
A matrix is a table of numbers with rows and columns. Stack all the passage vectors one under another, and the index becomes a matrix P. Each row is one passage, and each column is one dimension. Our index has 4 rows and 2 columns, which is called a 4 × 2 matrix:
Row, one passage
Column 1
Column 2
Dot product with query [0.8, 0.2]
Office hours page
0.9
0.1
0.74
Syllabus
0.1
0.9
0.26
Project page
0.2
0.8
0.32
FAQ page
1.6
1.4
1.56
The last column is the matrix-vector product P q. Its rule is: take the dot product of each row of the matrix with the vector q, and write the results one under another. A 4 × 2 matrix and a vector of 2 numbers give 4 numbers, here [0.74, 0.26, 0.32, 1.56]. The number of columns in the matrix must equal the number of entries in the vector, and that is the fixed-dimension condition.
The transpose and the slide's EQ(q)⊤EP(p)
In matrix language a vector is usually written as a column, a matrix with 2 rows and 1 column. The transpose, written ⊤, turns rows into columns, so a column vector becomes a row. A row (1 × 2) times a column (2 × 1) gives a 1 × 1 result, a single number, and that number is exactly the dot product:
[0.8 0.2] × 0.90.1 = 0.8 × 0.9 + 0.2 × 0.1 = 0.74
The EQ(q)⊤EP(p) on Stanford's DPR slide is this: the query encoder's vector becomes a row, and the passage encoder's vector stays a column. Lesson 4 writes the same thing as .
Normalize every row of the index first, and the query too. Then the same single product gives all four cosines: [0.99, 0.35, 0.47, 0.89]. Lesson 12 applies this same idea to a whole batch, as a product of two matrices.
At real scale, DPR's index has about 21 million rows and 768 columns. from the paper For one query this matrix-vector product is 21 million dot products, so 21 million × 768 = about 16 billion multiplications. Lesson 10 shows ways to make this faster.
8Stanford's table: five kinds of embeddings
Stanford's embeddings table lists five kinds of embeddings, and they differ in what one vector stands for. Retrieval uses the passage embeddings directly, because the index needs one vector per chunk. The other kinds show that a vector can also stand for a word, a token, an image or a node. Lesson 7 returns to them together with chunking.
Kind
Examples
One vector stands for
word2vec, GloVe
A word type
BERT, RoBERTa
A token in context
Sentence-BERT, DPR, E5
A chunk
CLIP
Text or an image, in a shared space
node2vec, GNN encoders
A node or a relation
Static word. Every has one vector, whatever the sentence. In Late days may not be applied and late afternoon office hours, late gets the same vector. The word2vec paper showed that vector(King) − vector(Man) + vector(Woman) is closest to vector(Queen). from the paper
Contextual. Each token's vector is built from the words around it, so the two lates get different vectors. BERT-base gives every token a vector of 768 numbers. For a whole passage, DPR takes the vector of a special token called [CLS], which is added at the start of every text. from the paper
Passage. One vector for a whole chunk, the embedding of this lesson. Sentence-BERT averages the token vectors, which is called . E5 reads both queries and passages with one encoder, and to tell them apart it puts query: or passage: in front of the text. from the paper
Multimodal. Texts and images get vectors in the same space, so their cosine can be computed. A photo of office hours written on a whiteboard and the text office hours schedule can get vectors close together. CLIP was trained on 400 million (image, text) pairs collected from the internet, and it compares images and texts by cosine. from the paper
Graph. A vector for a node or a relation in a network, such as TA, office hours and the runs relation between them in a course graph. node2vec walks randomly over the graph to produce lists of nodes, and learns from them as if they were sentences, with word2vec's Skip-gram method. from the paper GNN encoders also produce a vector for every node.
Why one vector per chunk
The Sentence-BERT paper gives the clearest arithmetic for this. Suppose you want the most similar pair among 10,000 sentences, and the model reads each pair together, as BERT does. Then there are 10,000 × 9,999 ÷ 2 = 49,995,000 pairs, about 50 million. According to the paper, this takes about 65 hours on a V100 GPU. from the paper
With one vector per sentence, the model runs only 10,000 times. In the paper this took about 5 seconds, and all the cosines then took about 0.01 seconds. from the paperThat is why retrieval builds each chunk's vector offline and, on a query, only computes dot products or cosines.
Why a rare error code gets lost in one vector
One vector per chunk is fast, but it has a cost. Suppose a passage's vector is the average of the vectors of its 20 tokens, that is, mean pooling. Then each token contributes only 1/20 of the average. Now take two passages that are identical except for one token: one says E-4012 and the other says E-4021. my numbers
Let the other 19 tokens average [0.5, 0.5]. The vector of E-4012 is [1, 0] and that of E-4021 is [0, 1], so the two codes have cosine 0.
The two codes pointed in completely different directions, yet the passages have cosine 0.995. So the query E-4012 upload failed will score both passages almost equally. BM25 from lesson 11 gives such a rare code a high , so it finds the right passage.
The DPR paper has a real example of this. For the question Who plays Thoros of Myr in Game of Thrones?, BM25's top passage contains Thoros of Myr word for word, along with the name of the actor who played him. DPR's top passage is about a Norwegian actor who played Thor Heyerdahl. The paper suggests that DPR's vector may lack the capacity to represent salient phrases that appear rarely. from the paper
The same paper has the opposite example too. For the question What is the body of water between England and Ireland?, DPR returns the passage about the Irish Sea, while BM25 returns a passage about cycling in which England and Ireland appear many times. from the paper This is the first point of Stanford's hybrid search slide. BM25 misses things said in different words, and dense retrieval misses rare . That is why lesson 14 runs both retrievers and merges their lists.
9Now read Stanford's slides yourself
Below are this lesson's three Stanford slides, unchanged. Under each slide are a few questions. Think of the answer first, then open it. If a symbol stops you, go back to the step named in the answer. If a slide looks small on a phone, click the image to open it at full size.
Stanford's slide: What Makes A Good Retriever? Embeddings. Below it, its table and last line, enlarged.
Read the embeddings slide
In the static word row and the passage row, what does one vector stand for? Use the late example to show the difference.
Show the answer
In static word embeddings each word type has one vector, so late has the same vector in both sentences. In the passage row one vector stands for a whole chunk, and these are the vectors a retrieval index stores. (Step 8)
The last line says each chunk's vector must be fixed-dimension. How many numbers does the long FAQ page's vector have, and why is this condition needed?
Show the answer
2 in our example and 768 in DPR, however long the chunk is. The dot product pairs up numbers by position. The index matrix needs as many columns as the query's vector has numbers. (Step 7)
The line says cosine or inner product. For our TA query, which passage comes first under each, and what happens after normalizing?
Show the answer
The FAQ page with 1.56 by inner product, and the office hours page with 0.99 by cosine. On normalized vectors the two give the same score, so the office hours page comes first under both. (Steps 3 and 6)
Stanford's slide: DPR, dense passage retrieval. Below it, the vector index, dot product and top-k part, enlarged.
Read the DPR slide
Write the EQ(q)⊤EP(p) from the dot product box with our query and the office hours page.
Show the answer
The query's vector becomes a row, and the passage's vector a column: [0.8 0.2] times the column [0.9, 0.1] = 0.8 × 0.9 + 0.2 × 0.1 = 0.74. (Step 7)
What does the vector index hold, and what does the arrow from it to the dot product box do on one query?
Show the answer
The index is the matrix of all passage vectors, built offline. When a query arrives, one matrix-vector product gives all the scores: [0.74, 0.26, 0.32, 1.56]. The top-k are the k largest of these, so for k = 2 the FAQ page and the office hours page. (Steps 4 and 7)
The slide uses the dot product as the score. Why didn't the DPR paper use cosine or distance?
Show the answer
All three are decomposable, so any of them would allow an offline index. In the paper's experiments L2 was comparable to the dot product, and both beat cosine. On NQ dev, top-20 was 78.1% with the dot product and 76.1% with L2, so the simplest one, the dot product, stayed. (Step 6)
Stanford's slide: Hybrid Search and Reciprocal Rank Fusion. Below it, its first point and formula, enlarged. Lesson 14 covers hybrid search and RRF in full.
Read the hybrid search slide
The first point says dense retrieval misses rare literals. In the E-4012 example, what cosine did the two passages get, and why?
Show the answer
0.995. The passage vector was the average of 20 tokens, so the code token contributed only 1/20 of it. That is why the difference between the two codes almost disappears in the average. (Step 8)
The other half of that point: BM25 misses paraphrase. Which query in this lesson is an example?
Show the answer
The TA query. It shares no word with the office hours page, so a score based on words gives every passage 0. The embedding retriever put the office hours page on top with 0.78. (Step 4)
The slide says: merge positions instead of raw scores. Why can't BM25 and cosine scores simply be added? With k = 60 in the formula, how much does a passage in first place on one list get?
Show the answer
Their scales differ. Cosine never goes above 1, while in lesson 11 BM25 gave the forum post 2.749. The dot product has no ceiling either. A passage in first place gets 1 ÷ (60 + 1) = 0.0164. Lesson 14 works through RRF in full. (Steps 5 and 8)
3 points from this lesson
A vector is a list of numbers with a fixed length, and two vectors are compared with the dot product or cosine similarity. Cosine is the cosine of the angle between the two arrows, so it stays between −1 and 1.
The dot product rewards a long vector, as with the FAQ page's 1.56. Cosine ignores length, and distance penalizes a long vector. On normalized vectors all three give the same order, and DPR kept the dot product.
The index is a matrix, and one matrix-vector product gives every passage its score. One vector per chunk makes search fast, but a rare error code can get lost in it. That is why lesson 14 combines BM25 with dense retrieval.
New terms in this lesson
Vector
A list of numbers with a fixed length, such as [0.9, 0.1]. With 2 numbers, it is an arrow on a plane.
Dimension
How many numbers a vector has. For example, [0.9, 0.1] has dimension 2, and BERT-base vectors have 768.
Dot product (inner product)
Multiply the numbers in the same position, then add them all. For example, [0.8, 0.2] and [0.9, 0.1]: 0.72 + 0.02 = 0.74.
∑ (sigma)
Keep adding over the range written below it. For example, the sum of aibi for i = 1 to 2 is a1b1 + a2b2.
Length
The length of the arrow: square each number, add the squares, take the square root. For example, [0.8, 0.2] has length √(0.64 + 0.04) = 0.82.
Cosine similarity
The dot product divided by the product of both lengths. It looks only at direction and stays between −1 and 1. For example, 0.74 ÷ (0.82 × 0.91) = 0.99.
Normalize
Divide every number in a vector by its length, so the length becomes 1 and the direction stays the same. For example, [0.8, 0.2] ÷ 0.82 = [0.97, 0.24].
Embedding
A text's vector, produced by a trained model. Texts saying similar things point in similar directions, such as “help from a TA” and “run by the TAs”.
Embedding model (encoder)
A model that reads a text and produces its fixed-length embedding. For example, turning the query into a vector like [0.8, 0.2].
Index
The list of passage embeddings built in advance. For example, 60,000 vectors for 60,000 passages.
Offline and online
Offline: once, before any query arrives, such as building the passage embeddings. Online: on every query, such as making the query's embedding.
Matrix and matrix-vector product
A table of numbers in rows and columns, and the dot product of each of its rows with a vector. For example, our 4 × 2 index times the query [0.8, 0.2] = [0.74, 0.26, 0.32, 1.56].
Decomposable similarity
A score that comes from a question vector and a passage vector, where the passage vector is built without seeing the question. Only then can an offline index exist. For example, the dot product, cosine and distance.