Probability tools: getting ready to read the RAG formula
This lesson builds five tools: conditional probability, the chain rule, marginalization, softmax and loss. Lesson 5's RAG formula is made of exactly these five. That formula combines two kinds of numbers. In lesson 1, after reading the queue log, the LLM gave Thursday a of 0.81. In lesson 3, the retriever gave the course's long FAQ page a of 1.56.
These two numbers do not combine directly. 1.56 cannot be a probability, because a probability is never larger than 1. And 0.81 is not the chance of the whole answer, only of its first token. The formula in lesson 5 combines them like this: p(y|x) ≈ ∑ p(z|x) · p(y|x, z), where p(z|x) ∝ exp(d(z)⊤q(x)), and training makes ∑ −log p(y|x) small. In step 5 you will run this whole formula by hand.
The ladder for this lesson
The ladder has 10 steps. The first 5 are the same as in the easy version, and steps 6 to 10 take you to the level of the Stanford slides. Under each step you can see what you will be able to do after it.
Conditional probability and the chain ruleWork out a probability like p(full | Tuesday) from counts, and build the probability of a whole answer from its tokens' probabilities.
What you need first: Lesson 1 (probability, token, next-token prediction, training) and Lesson 3 (vector, dot product, ∑). From step 6 on, a few words come from lesson 5, such as BERT, MIPS and η and θ, and clicking them opens their meaning. If you already know a step, skip to the next one. A word with a dotted underline and ▾ opens its definition when you click it.
1Conditional probability and the chain rule: a chance that depends on what you know
What comes after the “|” is already known
A conditional probability is the chance of something when something else is already known. It is written p(A | B) and read “the probability of A, given B”. Every probability in the RAG formula has this vertical bar “|” inside it, so start by seeing what it means with numbers.
Over the last 20 weeks there were 40 office hour sessions, 20 on Tuesday and 20 on Thursday. Call a session “full” if more than 10 students were waiting. my numbers
Full
Empty
Total
Tuesday
16
4
20
Thursday
2
18
20
Total
18
22
40
If you do not know the day, count all 40 sessions. 18 of them were full, so p(full) = 18 ÷ 40 = 0.45. Now suppose you know today is Tuesday. Then you count only the Tuesday row, where 16 of 20 were full, so p(full | Tuesday) = 16 ÷ 20 = 0.80. If you know it is Thursday, p(full | Thursday) = 2 ÷ 20 = 0.10. As soon as the day is known, the chance moves from 0.45 to 0.80 or to 0.10.
p(A | B) = cases with both A and Bcases with B
For Tuesday, the top is 16, the full Tuesdays, and the bottom is 20, all Tuesdays.
Next-token prediction from lesson 1 is also a conditional probability. There the prompt was known, and Thursday had probability 0.21. When the queue log was added to the prompt, what was known changed, and the same probability became 0.81.
You can now read both pieces of the formula. p(z|x) is the probability of passage z, given the question x. p(y|x, z) is the probability of the answer y, given both the question x and the passage z. Read the comma after the “|” as “and”.
The probability of a whole answer: the chain rule
An LLM writes one token at a time, and each token has its own probability. But p(y|x, z) in the formula is the probability of the whole answer y, such as Thursday morning. The rule for turning three token probabilities into one probability is already visible in the table above.
In the table, Tuesday and full happened together 16 times out of 40, which is 0.40. The same number also comes out in two steps. First take the chance of Tuesday: p(Tuesday) = 20 ÷ 40 = 0.5. Then take the chance of full, given Tuesday: p(full | Tuesday) = 0.8. Their product is 0.5 × 0.8 = 0.40.
p(A and B) = p(A) × p(B | A)
The probability of two things together is the probability of the first, times the probability of the second given the first. This rule is the . Lesson 12 has a different rule with the same name, about rates, so call this one the chain rule for probabilities.
For the tokens of an answer, this rule runs again and again. The first token is Thursday, with probability 0.21. Once Thursday is written, morning has probability 0.31, and then the full stop has 0.71. The probability of the whole answer is the product of the three. For the two answers from the widget in lesson 1, it looks like this:
Answer
Probability of each token
Probability of the whole answer
Thursday morning.
0.21 × 0.31 × 0.71
0.046
Wednesday afternoon.
0.34 × 0.46 × 0.71
0.111
For an answer of n tokens, the same product is written with . ∏ means multiply everything over the range written below it, just as ∑ means add everything. yi is the i-th token of the answer, and after the “|” in its probability come the question x and all the tokens before it:
p(y | x) = ∏i = 1 … n p(yi | x, y1 … yi−1)
In our answer n = 3: y1 = Thursday, y2 = morning and y3 = the full stop. In the probability of y2, the question and Thursday come after the “|”. The whole product is 0.21 × 0.31 × 0.71 = 0.046.
This product has a problem. Every probability is below 1, so the result shrinks with every multiplication. With 20 tokens of probability 0.5 each, the whole answer gets 0.520 = 0.00000095. The ln in step 4 solves exactly this problem.
Think first
An answer has 4 tokens with probabilities 0.9, 0.8, 0.5 and 0.9. What is the probability of the whole answer?
Show the answer
0.9 × 0.8 = 0.72, then 0.72 × 0.5 = 0.36, and then 0.36 × 0.9 = 0.324. The other three tokens are at 0.8 or above, yet the single 0.5 token halved the probability of the whole answer.
2Marginalization: when you do not know which case is true
Marginalization is the way to get a total probability when you do not know which case is true. Multiply each case's probability by the chance within that case, then add up the products. We will call a case's probability its weight.
This is exactly the situation in RAG. The retriever brings several passages, and the system does not know in advance which passage is right. With the queue log, the LLM will give Thursday 9am a high probability, and with the old forum post a low one. The system has to turn these into a single probability.
First look at the same situation with office hours, where the numbers are small. A student goes on Tuesday half the weeks and on Thursday the other half, so p(Tuesday) = 0.5 and p(Thursday) = 0.5. The chance of a full session on each day is known from step 1: 0.8 and 0.1. You just do not know which day the student will go today.
Case z
Weight p(z)
p(full | z)
Product
Tuesday
0.5
0.8
0.40
Thursday
0.5
0.1
?
Sum = p(full)
?
Work out the Thursday row
What is the Thursday product, and what is the total p(full)? If the student goes on Thursday 90% of the time, what is the total chance?
Show the answer
Thursday: 0.5 × 0.1 = 0.05. The sum is 0.40 + 0.05 = 0.45, the same number step 1 got by counting directly, 18 ÷ 40.
With Thursday 90% of the time: 0.1 × 0.8 + 0.9 × 0.1 = 0.08 + 0.09 = 0.17. The chance on each day stayed the same, only the weights changed, and the total went from 0.45 to 0.17.
This method is called . It is correct only when the cases do not overlap and their weights add up to 1. Both hold here, because on a given day the student goes either on Tuesday or on Thursday, and 0.5 + 0.5 = 1.
p(A) = ∑z p(z) × p(A | z)
z is a case, here Tuesday or Thursday. p(z) is its weight, 0.5 and 0.5, and p(A | z) is the chance of full in that case, 0.8 and 0.1. adds over all the cases, here 0.40 + 0.05, and the result is p(A) = 0.45.
The RAG formula in lesson 5 is exactly this. There a case z is a passage, and its weight is the retriever's confidence p(z|x). The probability of the answer with each passage is p(y|x, z), the product of token probabilities from step 1.
The RAG formula has instead of “=”, which means “approximately equal”. RAG does not add up every passage, only the retriever's passages, the K passages with the highest scores. The other passages have very small weights, so the sum stays nearly the same.
One condition is still open. Every weight must lie between 0 and 1, and all the weights must add up to 1. The retriever's scores are not like that, so step 3 turns them into weights.
3Softmax: weights from the retriever's scores
Softmax turns a list of scores into a list of probabilities. It has two steps: take the exp of each score, then divide each exp by the sum of all the exps. First see why both steps are needed.
In lesson 3 the question when can I get help from a TA had a vector. The retriever took the of that vector with each passage's vector, multiplying the numbers in the same positions and adding the results. For the office hours page that was 0.8 × 0.9 + 0.2 × 0.1 = 0.74. The other scores were: syllabus 0.26, project page 0.32 and the course's long FAQ page 1.56.
These are not probabilities, because 1.56 is larger than 1, and the four add up to 2.88, not 1. Dividing each score by the sum of the four does not always work either, because a dot product can also be negative. That would give a negative share, and a probability is never negative.
So softmax first turns every score into a positive number, and it uses for that. exp(s) = es, where e is a fixed number, about 2.718. For example, exp(1) = 2.718 and exp(2) = e × e = 7.389. exp(0) = 1, because any number raised to the power 0 gives 1.
For a negative score, exp is still positive: exp(−1) = 1 ÷ e = 0.368. Whatever the score, exp is always above 0, and a larger score gives a much larger exp. One rule of exp will be useful later: adding 1 to a score multiplies its exp by e, because ea+1 = ea × e. Check: exp(2) = exp(1) × e = 2.7183 × 2.7183 = 7.389.
Now see the two steps of softmax on the scores 2.0, 1.0 and 0.1: my scores
Score s
exp(s)
Arithmetic
Probability
2.0
7.389
7.389 ÷ 11.212
0.659
1.0
2.718
2.718 ÷ 11.212
0.242
0.1
1.105
1.105 ÷ 11.212
0.099
Sum
11.212
1.000
The three exps add up to 11.212, and dividing each exp by that gives 0.659, 0.242 and 0.099. All three lie between 0 and 1 and add up to 1, so they can serve as weights. This whole method is called :
pi = exp(si)∑j exp(sj)
On top is the exp of score si, such as exp(2.0) = 7.389 in the first row. Below is the sum of the exps of all the scores, 11.212. So p1 = 7.389 ÷ 11.212 = 0.659.
Think first
Add 1 to every score, giving 3.0, 2.0 and 1.1. Will the probabilities go up, go down, or stay the same?
Show the answer
They stay the same: 0.659, 0.242 and 0.099. Adding 1 to a score multiplies each exp by e, as when exp(2.0) = 7.389 becomes exp(3.0) = 20.09. The top and the bottom are both multiplied by e, and the division cancels it. So softmax looks only at the differences between scores.
The formula in lesson 5 writes softmax in a short form:
p(z|x) ∝ exp(d(z)⊤q(x))
means “is proportional to”. You first take the exp for each passage, then divide them all by one and the same number, chosen so that they add up to 1. That number is the sum of all the exps, so the softmax division is hidden inside ∝.
is the dot product of the passage's vector and the question's vector. d(z) is the vector of passage z, and q(x) is the vector of question x. The ⊤ is only a way of writing it, and the arithmetic is the same as in lesson 3. For example, the office hours page [0.9, 0.1] and the question [0.8, 0.2] give 0.74.
Now the 1.56 from the start of the lesson can become a weight too. Softmax over the four scores from lesson 3 gives these weights: FAQ page 0.50, office hours page 0.22, project page 0.14 and syllabus 0.14. my arithmetic Softmax does not change the order, because a larger score always gives a larger exp. So the FAQ page is still on top, and softmax does not fix the long-vector problem from lesson 3.
Run softmax yourself
Change the scores of the three passages. The probabilities always add up to 1, and even with a negative score every probability stays above 0.
Passage
Score
exp(score)
Probability
4ln and loss: one number for the error
A loss is a number that says how far the model is from the right answer, and training makes this number small. In RAG the loss is built from the probability of the right answer, using ln. ln also solves the tiny-number problem from step 1, so look at ln first.
The , ln x, is the power you must raise e to in order to get x. So ln undoes exp: ln(ex) = x. For example, e1 = 2.718, so ln 2.718 = 1. And e0 = 1, so ln 1 = 0.
The ln of a number below 1 is negative, because getting such a number needs a negative power of e. For example, ln 0.1 = −2.303, because e−2.303 = 1 ÷ e2.303 = 1 ÷ 10 = 0.1.
The most useful rule of ln is this: ln(a × b) = ln a + ln b, so the ln of a product is the sum of the two lns. It comes from the rule of exp, because powers add when you multiply: e2 × e3 = e5. In numbers, 7.389 × 20.09 = 148.4, and e5 is also 148.4. ln gives exactly these powers, so ln 148.4 = 5 = 2 + 3 = ln 7.389 + ln 20.09.
For Thursday morning., this rule turns the product of three probabilities into the sum of three lns:
Check: e−3.074 = 0.046, the same product as before. For an answer of 20 tokens, instead of a tiny number like 0.520 = 0.00000095, you add ln 0.5 twenty times: 20 × (−0.693) = −13.86. my arithmetic
Now build the number for the error: = −ln p, where p is the probability the model gave the right answer. The minus sign in front makes the loss positive, because every probability below 1 has a negative ln. The table below shows how the loss grows as p falls:
Probability of the right answer
Loss = −ln p
Meaning
1
0
Fully sure and right. The loss cannot go lower.
0.9
0.105
Good
0.5
0.693
Half sure
0.1
2.303
Mostly wrong
0.01
4.605
Very wrong
At p = 0.9 the loss is 0.105, but at p = 0.1 it is 2.303, more than 20 times as much.
The probability the model gave the right answer is called the . The loss is minus the ln of it, so this loss is called the . If Thursday morning. is the right answer, its loss is −(−3.074) = 3.074.
In lesson 1, training changed the parameters a little. As a result, the probability of Thursday after Office hours are on went from 0.10 to 0.12. The loss fell from −ln 0.10 = 2.303 to −ln 0.12 = 2.120, a drop of 2.303 − 2.120 = 0.183.
Training data has many examples. Example j has a question xj and its right answer yj, and the total loss is the sum of the losses over all examples:
total loss = ∑j − log p(yj | xj)
Stanford's slide and the formula in lesson 5 write log, and they mean this ln.
The tells training which way, and by how much, to change each parameter so that the total loss goes down. In the Thursday example, training changed the parameters in the direction that raised the probability from 0.10 to 0.12 and lowered the loss by 0.183. How the gradient is worked out from numbers comes in lesson 12.
5The whole RAG formula by hand
Now the tools work together: softmax gives each passage a weight, marginalization gives the answer's total probability, and −ln gives its loss. Take the question from the start again: Which office hour is least crowded? The retriever brings two passages: the queue log and the old forum post. Their dot product scores are 1.0 and 0.0.
The LLM gives the right answer Thursday 9am a probability of 0.9 with the queue log, and 0.1 with the forum post. my numbers The whole calculation takes three steps:
Softmax (step 3): e1.0 = 2.718 and e0 = 1, which add up to 3.718. The queue log's weight is 2.718 ÷ 3.718 = 0.731, and the forum post's is 1 ÷ 3.718 = 0.269.
So the system is 0.685 sure that the answer is Thursday 9am, and the loss on this question is 0.378. If the retriever gives the queue log a higher score, its weight rises above 0.731, 0.685 rises too, and the loss falls. The formula holds the same three steps:
The dot product of the passage's and the question's vectors, here 1.0 and 0.0.
p(z|x)
The softmax weight, here 0.731 and 0.269.
p(y|x, z)
The probability of the whole answer with that passage, here 0.9 and 0.1.
∑ and ≈
The weighted sum over K = 2 passages, here 0.685.
−log p(y|x)
The loss for this question, here 0.378.
Stanford's slide: Deep Dive into RAG. At the top are the retriever p(z|x) and the p(y|x, z), the LLM that writes the answer. In the middle is this formula. The red note says to weight each passage's answer by how much the retriever trusts that passage. The RAG-Sequence and RAG-Token lines at the bottom come in lesson 5. Click the image to open it at full size.
Think first
Suppose the retriever trusts the wrong passage: the queue log gets score 0.0, and the forum post 1.0. The LLM's numbers stay at 0.9 and 0.1. What are the total probability of Thursday 9am and the loss now?
Show the answer
The weights flip: the queue log gets 0.269 and the forum post 0.731. Total probability = 0.269 × 0.9 + 0.731 × 0.1 = 0.242 + 0.073 = 0.315, and loss = −ln 0.315 = 1.155.
The LLM changed nothing, yet the probability fell from 0.685 to 0.315. Wrong weights drag the whole answer down, and in lesson 5 training fixes exactly this mistake.
From here: the Stanford level
The 5 steps above are the same as in the easy version. The 5 steps below take you to the level of the Stanford slides: first every symbol in the slide's formula and the remaining rules of probability, then the remaining rules of exp and ln. After that comes softmax in full, then the training loss and its rates in numbers, and finally you read the slides yourself. Every piece of maths is explained where it first appears.
6Every symbol in the slide's formula, and the remaining rules of probability
Up to step 5 the formula ran on one question and two passages. Stanford's slide writes the same formula in symbols, so that it applies to every question and every passage:
p(y|x) ≈ ∑z ∈ top−K p(z|x) p(y|x, z)
The table below ties each symbol on the slide to a number from our example:
Symbol
Meaning
In our example
x
The question, which the slide calls the query
Which office hour is least crowded?
y
The answer
Thursday 9am
z
A passage
The queue log or the forum post
p(z|x)
Retriever: the weight of passage z, given the question
0.731 and 0.269
p(y|x, z)
Generator: the probability of the answer, given the question and the passage
0.9 and 0.1
p(z|x) p(y|x, z)
The product of the two. In maths, writing two things side by side means multiplying them.
0.658 and 0.027
z ∈ top−K
The range under the ∑: every z among the retriever's top-K passages. ∈ means “belongs to”.
K = 2: the queue log and the forum post
≈ and p(y|x)
The answer's total probability, only approximately equal because the other passages are left out
0.685
The generator's product, in the paper's symbols
The slide says the generator writes the answer one token at a time. That is the chain rule from step 1, and the RAG paper writes it out in full: from the paper
pθ(y | x, z) = ∏i = 1 … N pθ(yi | x, z, y1:i−1)
Here N is the number of tokens in the answer, and y1:i−1 means token 1 through token i−1, all the tokens before token i. In Thursday morning., for i = 2 this is y1:1 = Thursday, and for i = 1 there is no token before. The paper writes the generator as pθ and the retriever as pη, and come in lesson 5.
The chain rule works the same way for three or more things: p(A and B and C) = p(A) × p(B | A) × p(C | A and B). In the probability of each new thing, everything before it comes after the “|”. The generator's product is exactly this, except that the question x and the passage z are also known for every token.
p(A|B) and p(B|A) are different
Swap the two sides of the “|” and the probability changes. In the table from step 1, p(full | Tuesday) = 16 ÷ 20 = 0.80. But p(Tuesday | full) counts the full sessions: 16 of the 18 full sessions were on Tuesday, so it is 16 ÷ 18 = 0.89.
The top number, 16, is the same in both, but the bottom is different. So when you read a formula, check what comes after the “|”. p(y|x, z) is the probability of the answer when the passage is known, not the probability of the passage when the answer is known.
How close the ≈ is
Marginalization requires adding up all the cases. RAG adds only the top-K passages, and softmax also runs only over those K scores. To see the effect, add a third passage to the corpus: the Homework 1 announcement, with score −1.0, with which the LLM gives Thursday 9am a probability of 0.1. my numbers
The three passages have exps 2.718, 1 and 0.368, which add up to 4.086. So the softmax weights are 0.665, 0.245 and 0.090. The full sum is then 0.665 × 0.9 + 0.245 × 0.1 + 0.090 × 0.1 = 0.599 + 0.024 + 0.009 = 0.632, while over the top 2, in step 5, the sum was 0.685.
Homework 1 announcement
Sum over all three passages
Sum over the top 2
Score −1.0weight 0.090
0.632
0.685
Score −3.0weight 0.013
0.677
0.685
If the Homework score is −3.0, its weight is only 0.013, and the gap between the two sums shrinks from 0.053 to 0.008. So the smaller the weights of the passages left out, the closer the ≈. The top-2 sum is a little larger, because softmax shares the left-out passage's weight between the other two, and with that passage the answer's probability was only 0.1.
The paper says the retriever returns a distribution truncated to the top K, that is, a list of probabilities for only K passages, adding up to 1. During training K is 5 or 10. from the paper Out of about 21 million Wikipedia passages only 5 or 10 are added up, so the ≈ is close only when all the other passages have very small weights.
7The remaining rules of exp and ln
Steps 3 and 4 gave a few rules of exp and ln. The slides and the later lessons need a few more, and all of them follow from the fact that ln undoes exp. Each rule below comes with a check in numbers:
Rule
Check
1. ea+b = ea × eb
e1.5 = e1 × e0.5 = 2.718 × 1.649 = 4.482
2. e−a = 1 ÷ ea
e−1 = 1 ÷ 2.718 = 0.368
3. ln(ex) = x, and eln x = x
ln 7.389 = 2, and eln 0.685 = e−0.378 = 0.685
4. ln(a × b) = ln a + ln b
ln(0.21 × 0.31) = (−1.561) + (−1.171) = −2.732
5. ln(a ÷ b) = ln a − ln b
ln(7.3891 ÷ 2.7183) = 2 − 1 = 1, and indeed 7.3891 ÷ 2.7183 = 2.7183, which is e
6. ln(an) = n × ln a
ln(0.520) = 20 × (−0.693) = −13.86
Rule 5 follows from rule 4. a ÷ b means a × (1 ÷ b), and by rules 2 and 3, ln(1 ÷ b) = −ln b, because e−ln b = 1 ÷ eln b = 1 ÷ b. Rule 6 is rule 4 applied n times, as with the 20-token answer.
The ln of a softmax share
Rules 3 and 5 together make the ln of a softmax share simple. The share is es ÷ Z, where Z is the sum of all the exps, so ln(es ÷ Z) = s − ln Z. In step 5 the queue log's score was 1.0 and Z = 3.718. So −ln 0.731 = −1.0 + ln 3.718 = −1.0 + 1.313 = 0.313. The contrastive loss in lesson 12 starts from exactly this form.
The ln of a sum does not split
There is no such rule for ln(a + b), and in general ln(a + b) is not equal to ln a + ln b. Take the sum from step 5: −ln(0.658 + 0.027) = −ln 0.685 = 0.378. Taking −ln of each part separately and adding gives 0.419 + 3.612 = 4.031, a completely different number.
That is why the slide calls the loss the negative marginal log-likelihood: the marginalization over passages comes first, and the log is applied to the whole sum. The loss cannot be split passage by passage. It means a single loss holds both the retriever's weights and the generator's probabilities, which step 9 shows.
log10 and ln, and −ln 0
The slide just says log. In machine learning papers this almost always means the natural log, ln. At school, log often meant log10, the power you raise 10 to: log10 0.1 = −1, while ln 0.1 = −2.303.
The two differ by a fixed factor: the ln of any number is 2.303 times its log10, because ln 10 = 2.303. For example, the loss from step 5 would be 0.378 ÷ 2.303 = 0.164 with log10. Every example's loss would be divided by the same number, so which parameters are best does not change.
The other limit is at 0. ex is never 0, so ln 0 has no value. The closer p gets to 0, the larger −ln p grows: −ln 0.001 = 6.908, and −ln 0.000001 = 13.816. A probability that comes out of softmax is never exactly 0, because exp is always above 0, so such a loss can be very large but never infinite.
8Softmax in full: the normalizer, column vectors and top-k
This step covers every part of it: first the remaining properties of softmax, then the division hidden behind ∝, then the bold 𝐝 and 𝐪 and the ⊤, and finally top-k.
Negative scores and scale
A dot product can be negative, and softmax handles that too. The scores 1, 0 and −1 add up to 0, and you cannot divide by 0. But their exps 2.718, 1 and 0.368 add up to 4.086, giving probabilities 0.665, 0.245 and 0.090.
In step 3, adding 1 to every score changed nothing. That holds for any number c, because by rule 1, es+c = es × ec, and ec cancels between the top and the bottom. Multiplying the scores does make a difference. Multiply 2.0, 1.0 and 0.1 by 2, and at 4.0, 2.0 and 0.2 the probabilities change from 0.659, 0.242, 0.099 to 0.864, 0.117 and 0.019. The gaps doubled, so the top score's share grew.
In a dot product, this scale comes from the length of the vectors. In lesson 3 the FAQ page's vector was [1.6, 1.4], twice [0.8, 0.7]. So its score was doubled too: 1.56 = 2 × 0.78.
The division behind ∝: the normalizer Z
a ∝ b means a = c × b, where c is the same number for every passage. In softmax c = 1 ÷ Z, where Z is the sum of all the exps. This Z is called the normalizer, because it is what makes the probabilities add up to 1.
In step 5, Z = 2.718 + 1 = 3.718, so c = 1 ÷ 3.718 = 0.269. The queue log's weight is 2.718 × 0.269 = 0.731, and the forum post's is 1 × 0.269 = 0.269. Z depends on the question but not on any one passage, so the ratios between passages show without it, and the formula hides it inside ∝. In RAG this sum runs over the top-K passages only, as step 6 showed.
Bold 𝐝 and 𝐪, column vectors and ⊤
On the slide, 𝐝 and 𝐪 are bold. In maths a bold letter means a vector, not a single number. 𝐝(z) is the vector of passage z, and 𝐪(x) is the vector of question x.
Maths usually writes a vector standing up, with its numbers one above the other, and calls it a column vector. ⊤ means transpose: laying the standing vector down, turning the column into a row. With the vectors of the office hours page and the question, it looks like this:
The rule for a row times a column is: multiply the row's first number by the column's first number, the second by the second, and add everything up. That is the dot product, so 𝐝⊤𝐪 is a way of writing the dot product. Stack the rows of many passages one under another, and a single multiplication gives all their scores. That matrix product comes in lesson 12.
BERTd and BERTq are two separate . The small d says the first one reads passages, because the paper calls a passage a document. The small q says the second one reads the query, the question. Both are built from -base. from the paper
Why MIPS is enough for top-k(p(·|x))
The slide's second line is top-k(p(·|x)). The “·” marks an empty slot, which each passage fills in turn. So p(·|x) is the full list of weights of all passages for question x, and top-k is the k passages with the largest weights in it.
Finding these k passages does not need softmax, because exp is an increasing function: a larger input always gives a larger output. Dividing by Z is also the same for every passage, so the weights come in the same order as the scores. The lesson 3 scores 1.56, 0.74, 0.32 and 0.26 give weights 0.499, 0.220, 0.145 and 0.136, in exactly the same order.
So the passages with the k largest weights are the ones with the largest dot products. Another name for the dot product is the inner product, so this search is called Maximum Inner Product Search, or . According to the paper, MIPS finds an approximate top k without looking at every passage, so when the number of passages doubles, the search time grows by less than double. from the paper How that works is in lesson 10.
9Negative marginal log-likelihood, and the loss's rates in numbers
Stanford's second slide says: Jointly train the retriever and generator to minimize the negative marginal log-likelihood ∑j −log p(yj|xj). In this step, first read that name word by word, then see how a single loss tells both models which way to change.
The four words of the name
Word
Meaning
In step 5
likelihood
The probability of the right answer, p(y|x)
0.685
marginal
This probability comes from marginalization over passages
0.658 + 0.027
log-likelihood
Its ln
ln 0.685 = −0.378
negative
A minus sign in front, so that the loss is positive
0.378
So the loss for each example is −ln of a sum, and step 7 showed that this sum does not split passage by passage.
The loss over many examples
∑j adds over all training examples. Suppose a second example is the question Where are office hours held?, and the system gives its right answer a probability of 0.5. my number The total loss over the two examples is then 0.378 + 0.693 = 1.071.
The same number also comes out another way. The two questions do not depend on each other, so in the chain rule the probability of the second answer does not change with the first. The probability that both answers are right is then 0.685 × 0.5 = 0.3425, and −ln 0.3425 = 1.071, because the rule of ln turns a product into a sum. So making the total loss small means making the probability of the whole training data being right large.
Training makes −ln p small instead of making p large, but the best parameters stay the same, because ln is also an increasing function. The larger p is, the smaller −ln p is, so both have the same best point. The advantage of −ln is that products turn into sums and no tiny numbers appear.
A rate: how much the loss changes
Step 4 gave the meaning of the gradient: which way to change each parameter so that the loss goes down. That direction comes from a rate. Change the input a little, say by 0.01, see how much the loss changed, and divide that change by 0.01. First look at this on the loss −ln p itself:
p
Change in loss when p rises by 0.01
Change ÷ 0.01
0.10
−0.0953
−9.53
0.50
−0.0198
−1.98
0.90
−0.0110
−1.10
Take an even smaller change than 0.01, and these rates settle at −10, −2 and −1.11, that is, −1 ÷ p. The rate is largest for an answer with a small probability, so that is where training pushes hardest. In lesson 1, p went from 0.10 to 0.12 and the loss fell by 0.183. The rate over that whole step is −0.183 ÷ 0.02 = −9.15, which lies between the rate −10 near 0.10 and the rate −8.33 near 0.12.
Jointly train: one loss, rates for both models
The loss of 0.378 from step 5 depends on four numbers: the retriever's two scores, 1.0 and 0.0, and the generator's two probabilities, 0.9 and 0.1. Raise each one by 0.01 in turn, keeping the other three fixed: my arithmetic
What rose by 0.01
Change in loss
Change ÷ 0.01
Retriever: the queue log's score
−0.00229
−0.229
Retriever: the forum post's score
+0.00230
+0.230
Generator: 0.9 with the queue log
−0.01062
−1.062
Generator: 0.1 with the forum post
−0.00392
−0.392
The retriever's two rates say the loss falls when the queue log's score rises, and rises when the forum post's score rises. Together they make −0.229 + 0.230 = 0.001, nearly 0, because if both scores rise equally, the softmax weights do not change.
Both of the generator's rates are negative, so whichever passage it is, a higher probability for the right answer lowers the loss. The queue log's rate is the larger one: 1.062 ÷ 0.392 = 2.71, close to 0.731 ÷ 0.269 = 2.72. So each of the generator's probabilities counts as much as its passage's weight.
A single loss gives both models their direction, and that is what “jointly train” on the slide means. In real models these four numbers are themselves produced by hundreds of millions of parameters. Each parameter has a rate like these, and the list of all these rates is the gradient. Training nudges every parameter a little against its rate.
These rates appear with the symbols ∂ and ∇ in lesson 12. The rule for which passage's score goes up and which goes down is in lesson 5. The paper says three more things about training: from the paper
The training data holds only pairs of a question and an answer, (xj, yj). Which passage is right is written nowhere in the data.
The paper changes the parameters in small steps using the gradient, with a method called Adam.
The passage encoder BERTd and the index stay fixed. Training changes only the parameters of BERTq and of the generator BART, because changing the passage encoder would mean rebuilding the index.
10Now read Stanford's slides yourself
Below are Stanford's three slides, unchanged, each followed by its formula enlarged. Under each slide are a few questions. Think of the answer first, then open it. If a symbol stops you, go back to the step named in the answer. If a slide looks small on a phone, click the image to open it at full size.
Stanford's slide: Deep Dive into RAG, and below it the formula, enlarged.
Read the RAG formula slide
Under the ∑ it says z∈top−K, and there is ≈ instead of an equals sign. What do the two mean, and how far off was the ≈ in our example?
Show the answer
The ∑ runs only over the retriever's top-K passages, and ∈ means “belongs to”. The other passages are not added, hence the ≈. Over the top 2 the sum was 0.685, and adding the Homework 1 announcement gives 0.632, a gap of 0.053. (Steps 2 and 6)
The generator writes the answer one token at a time. How does p(y|x, z) become a single number then?
Show the answer
By the chain rule. Take each token's probability, given the question, the passage and the earlier tokens, and multiply them all. The paper writes this as ∏ pθ(yi | x, z, y1:i−1), for example 0.21 × 0.31 × 0.71 = 0.046. (Steps 1 and 6)
The bottom of the slide mentions RAG-Sequence and RAG-Token. Which of the two does the formula above belong to?
Show the answer
RAG-Sequence. There one passage produces the whole answer, so p(y|x, z) is the product over all the answer's tokens, and the weighted sum over passages comes after. from the paper RAG-Token can use a different passage for each token, and its formula is in lesson 5.
Stanford's slide: the same two lines, with the training loss in a third line. Below, that line enlarged.
Read the training slide
Where does each word of “negative marginal log-likelihood” sit in ∑j −log p(yj|xj)?
Show the answer
The likelihood is p(yj|xj), the probability of the right answer. Marginal means this probability is a sum over passages. Log is its ln, negative is the minus in front, and ∑j adds over all examples. Over two examples this was 0.378 + 0.693 = 1.071. (Steps 4 and 9)
Why can't this loss be split into a separate loss for each passage?
Show the answer
Because the log is applied to the whole sum, and ln(a + b) does not split. −ln(0.658 + 0.027) = 0.378, while the separate −ln values add up to 4.031. (Step 7)
What does “jointly train” look like in our numbers?
Show the answer
The same loss has a rate for the retriever's scores and for the generator's probabilities. Raise the queue log's score by 0.01 and the loss falls by 0.00229. Raise the generator's 0.9 to 0.91 and it falls by 0.01062. In the paper the passage encoder and the index stay fixed, while the query encoder and the generator change. (Step 9)
Stanford's slide: Retriever: DPR, and below it the formula and the top-k line, enlarged.
Read the retriever slide
Which division is hidden behind ∝? What was that number in the example from step 5?
Show the answer
Division by Z, the sum of all the exps, which is called the normalizer. In step 5, Z = 2.718 + 1 = 3.718, and the queue log's weight was 2.718 ÷ 3.718 = 0.731. (Steps 3 and 8)
What do the bold 𝐝 and 𝐪, the ⊤, and the small d and q in BERTd and BERTq tell you?
Show the answer
Bold means a vector. ⊤ turns a column vector into a row, and a row times a column is the dot product: [0.9, 0.1] and [0.8, 0.2] give 0.74. BERTd is the encoder for passages and BERTq the one for questions, so these are two separate encoders. (Step 8)
Why is there no need to run softmax over all passages to find top-k(p(·|x)), and why is this search called MIPS?
Show the answer
exp is an increasing function, and the division by Z is the same for all passages, so the weights come in the same order as the dot products. Finding the passages with the k largest dot products is therefore enough, and another name for the dot product is the inner product. (Step 8)
3 points from this lesson
p(A|B) means “the chance of A, given B”, and it is different from p(B|A). By the chain rule, the probability of a whole answer is the product of each token's conditional probability.
The RAG formula is marginalization over the top-K passages. The ≈ is close only when the passages left out have small weights.
Softmax builds weights with exp and the normalizer Z, and keeps the order, so MIPS is enough for the top k. The loss is −ln of a sum, which does not split, so a single loss gives both the retriever and the generator their rates.
New terms in this lesson
Conditional probability, p(A|B)
The probability of A when B is known. Read “|” as “given”. For example, p(full | Tuesday) = 16 ÷ 20 = 0.80.
Chain rule (for probabilities)
p(A and B) = p(A) × p(B|A). With many steps, multiply them all. For example, p(Tuesday) × p(full | Tuesday) = 0.5 × 0.8 = 0.40. The chain rule for rates in lesson 12 is a different rule.
∏ (capital pi)
Multiply everything over the range written below it. For example, for three tokens, 0.21 × 0.31 × 0.71 = 0.046.
Marginalization
Each case's weight times the probability within that case, added up over all cases. For example, 0.5 × 0.8 + 0.5 × 0.1 = 0.45.
≈
Approximately equal. The RAG formula adds up only the top-K passages, so the sum is only approximately equal.
exp and e
exp(s) = es, where e is about 2.718. Always above 0, and ea+1 = ea × e. For example, exp(1) = 2.718, exp(0) = 1, exp(−1) = 0.368.
Softmax
The exp of each score, divided by the sum of all the exps. It turns scores into probabilities. For example, 2.0, 1.0, 0.1 become 0.659, 0.242, 0.099.
∝
Is proportional to. At the end, divide everything by one number so that the total is 1. For example, exps 2.718 and 1 become 0.731 and 0.269.
d⊤q
A way of writing the dot product of vectors d and q. For example, the office hours page [0.9, 0.1] and the question [0.8, 0.2] give 0.74.
Natural log, ln
The power you raise e to in order to get x. ln(ex) = x, and ln(a × b) = ln a + ln b. For example, ln(0.1) = −2.303.
Loss
A number for how wrong the model is. Training makes it small. For example, if the right answer has probability 0.5, the loss is −ln 0.5 = 0.693.
Likelihood
The probability the model gave the correct data. For example, if it gave the right answer 0.12, the likelihood is 0.12.
Negative log-likelihood
−ln p(right answer). The most common loss. For example, at p = 0.12, −ln 0.12 = 2.120.
Gradient
The calculation of which way, and how much, to change each parameter so that the loss goes down. For example, in lesson 1 training moved the parameters in the direction that raised Thursday's probability from 0.10 to 0.12 and lowered the loss from 2.303 to 2.120.
Normalizer, Z
In softmax, the sum of all the exps. Dividing by it makes the weights add up to 1, and it is the division hidden behind ∝. For example, for exps 2.718 and 1, Z = 3.718.
Column vector and ⊤
A column vector has its numbers written one above the other, and ⊤ (transpose) turns it into a row. A row times a column is the dot product. For example, the row [0.9, 0.1] times the column [0.8, 0.2] = 0.74.
Increasing function
A function where a larger input always gives a larger output, so the order is kept. exp and ln are such functions. For example, scores 1.56 and 0.74 give weights 0.499 and 0.220, in the same order.
Negative marginal log-likelihood
−ln of the right answer's probability, when that probability comes from marginalization over passages, added up over all examples. RAG's training loss. For example, over two examples 0.378 + 0.693 = 1.071.