Lesson 5 · Easy version

The RAG model: retriever and generator together

RAG is a model made of two models. The retriever brings passages for a question and gives each one a weight, and the generator writes the answer with each passage. Both are trained together with one loss, and that training needs only question-answer pairs. RAG stands for Retrieval-Augmented Generation, a model published by Lewis and colleagues in 2020.

The need for it showed up in the last question of lesson 4. There the retriever gave the old forum post a weight of 0.731 and the queue log only 0.269. The generator's numbers stayed the same, yet the probability of the right answer, Thursday 9am, fell from 0.685 to 0.315. The mistake is the retriever's, but the training data holds only questions and answers. Nothing in it says the answer came from the queue log, and RAG teaches the retriever anyway.

The ladder for this lesson

The ladder has 5 steps. Under each step you can see what you will be able to do after it.

  1. RAG's two models: retriever and generatorSay what each part does, and read Stanford's RAG slide.
  2. Marginalization: no passage label neededWork out an answer's probability by hand, and explain why training needs no passage label.
  3. RAG-Sequence and RAG-TokenCompute the probability both ways, and say when the two give different results.
  4. Training the retriever and the generator with one lossSay which passage's score training raises, and which parts change.
  5. Results on NQRead the paper's main table, and see why lesson 6 is needed.

What you need first: Lesson 2 (retriever, top-K), Lesson 3 (vector, dot product, encoder, index) and Lesson 4 (marginalization, softmax, ∏, loss). If you already know a step, skip to the next one. A word with a dotted underline and ▾ opens its definition when you click it.

1RAG's two models: retriever and generator

RAG contains two models. The retriever pη(z|x) gives every passage z a weight, and the generator pθ(y|x, z) reads the question x and one passage and writes the answer y. These are the same two probabilities that appeared in the formula of lesson 4. The small η (eta) and θ (theta) written underneath name the two models' , the numbers that training changes.

The retriever: DPR

RAG's retriever is a model called DPR, whose own training comes in lesson 12. It has two , both built from BERT, an encoder model Google released in 2018. The query encoder makes the question's vector q(x), and the passage encoder makes a vector d(z) for every passage z.

A passage's score is the of these two vectors, written d(z)⊤q(x). Then turns the scores into weights that add up to 1. In lesson 4, scores 1.0 and 0.0 became weights 0.731 and 0.269. Stanford's slide writes the same thing like this:

pη(z|x) ∝ exp(d(z)⊤q(x))

Here means: take each passage's exp, then divide them all by one and the same number so that they add up to 1.

There are a great many passages. In the paper, Wikipedia is cut into about 21 million passages of 100 words each. from the paper Their vectors are built once, before any question arrives, and kept in an . When a question arrives, only the query encoder runs, and then a search pulls out the passages with the K largest dot products. This search is called MIPS, maximum inner product search, because inner product is another name for the dot product.

The generator: BART

The generator reads the question and one passage and writes the answer, one token at a time, like the LLM in lesson 1. RAG's generator is , an encoder-decoder model. Its encoder reads the input text and turns it into vectors, and its decoder looks at those vectors while it writes the answer token by token.

BART reads a single text, so the question and the passage are joined end to end into one text. Joining two texts like this is called concatenating them. With the queue log, the input and the output look like this: my example

Input: question: Which office hour is least crowded? passage: Queue log, last 3 weeks: Tuesday 4pm office hour averaged 14 students waiting; Thursday 9am office hour averaged 3.
Output: Thursday 9am
Question xWhich office hour is least crowded?
Query encoder q(x)The question's vector
MIPSDot product with every d(z) in the index, top-K
z1 … zKA weight for each, from softmax
Generator pθRuns once per z
MarginalizeAdd up to get p(y|x)
RAG's whole path for one question. The first four boxes are the retriever, and the last two are the generator. The Marginalize box comes in step 2.

This design has a cost. The generator runs separately with each passage, so with K = 5 it runs 5 times for one question.

Stanford slide: the whole RAG design, retriever and generator
Stanford's slide: the RAG design. The green box is the retriever pη, holding the query encoder q, MIPS and the document index d(z). The blue box is the generator pθ. The dashed arrow at the top is training, which comes in step 4. Click the image to open it at full size.

2Marginalization: no passage label needed

RAG treats the passage as a hidden choice, and gets the answer's probability as a weighted sum over all the passages it brought. A hidden choice like this is called a latent variable. The answer came from some passage, but the data does not say which one.

Teaching the retriever directly would need the right passage written next to every question. Nobody picks passages by hand for millions of questions. from lesson 4 is the method for exactly this situation: multiply each case's probability by its weight, then add everything up. There it was 0.5 × 0.8 + 0.5 × 0.1 = 0.45.

Now take the question Which office hour is least crowded? The retriever brought 3 passages, so K = 3, and their scores are 1.8, 1.1 and 0.0. In softmax, e1.8 = 6.05, e1.1 = 3.00 and e0 = 1, which add up to 10.05. So the queue log's weight is 6.05 ÷ 10.05 = 0.60, and the other two get 0.30 and 0.10 the same way. my numbers

With each passage, the generator gives Thursday 9am a probability. Each row multiplies the weight by the generator's probability:

Passage zWeight p(z|x)p(“Thu 9am” | x, z)Product
Queue logThursday 9am averages 3 students0.60.90.54
Office hours pageTuesday 4pm and Thursday 9am, but not which one is quiet0.30.5?
Old forum postmidweek afternoons are usually the least crowded0.10.10.01
Sum = p(“Thu 9am” | x)?

Work out the second row

The office hours page has weight 0.3, and with it the generator gives 0.5. What is the product, and what is the total probability of Thursday 9am?

Show the answer

0.3 × 0.5 = 0.15. The total is 0.54 + 0.15 + 0.01 = 0.70.

For Wednesday afternoon the generator's numbers are 0.05, 0.2 and 0.8, because the forum post talks about midweek afternoons. Its sum is 0.6 × 0.05 + 0.3 × 0.2 + 0.1 × 0.8 = 0.03 + 0.06 + 0.08 = 0.17. So RAG picks Thursday 9am, 0.70 against 0.17.

In symbols, the same arithmetic reads:

p(y|x) ≈ ∑z ∈ top-K pη(z|x) · pθ(y|x, z)

pη(z|x) are the weights (0.6, 0.3, 0.1), pθ(y|x, z) are the generator's numbers (0.9, 0.5, 0.1), and ∑ adds up their products (0.70). The sum runs only over the top-K passages, not over the other 21 million, which is why the formula has instead of “=”.

Nowhere in this calculation does it say that the right passage is the queue log. The formula needs only the question x and the answer y, and the passage stays hidden inside the sum. So question-answer pairs are enough to train RAG.

Change the retriever's scores

The sliders are the retriever's scores, and the generator's numbers stay fixed. The widget uses exact weights, so it shows 0.701 below instead of 0.70. The last column comes in step 4.

1.8 1.1 0.0
Passagep(z|x)“Thu 9am”“Wed”Training

Now press “Broken retriever”. The retriever now gives the forum post a weight of 0.69, Wednesday afternoon climbs to 0.60, and Thursday 9am drops to 0.27. The generator's numbers are unchanged, so the whole difference comes from the retriever's weights. So RAG's answer depends on the retriever's weights just as much as on the generator's numbers.

The sum also has a limit. If the right passage never made it into the top K, it is not in the sum, and training cannot raise its weight. Lesson 2 called this the .

3RAG-Sequence and RAG-Token: when the weighted sum happens

RAG-Sequence writes the whole answer with each passage, then takes the weighted sum. RAG-Token takes a separate weighted sum at every token of the answer. So far the whole answer was one piece, but the generator writes it one token at a time. When different parts of the answer sit in different passages, the two methods give different results.

The probability of a whole answer is the product of its tokens' probabilities, which is the from lesson 4. If Thursday has probability 0.9, and then Zoom has 0.2, the whole answer has 0.9 × 0.2 = 0.18.

Now take the question Least crowded office hour, and where is it? The answer has two tokens: Thursday and Zoom. The queue log says Thursday is the quietest but not where it is held. The office hours page says Thursday 9am on Zoom but not which slot is quiet. my numbers

PassageWeightp(Thursday | x, z)p(Zoom | x, z, Thursday)
Queue log0.60.90.2
Office hours page0.40.50.9

RAG-Sequence first writes the whole answer with each passage, then takes the weighted sum:

  1. Whole answer with the queue log: 0.9 × 0.2 = 0.18.
  2. Whole answer with the office hours page: 0.5 × 0.9 = 0.45.
  3. Weighted sum: 0.6 × 0.18 + 0.4 × 0.45 = 0.108 + 0.180 = 0.288.

The slide puts it this way: pick one passage, generate the whole answer with it.

RAG-Token first takes the weighted sum at each token, then multiplies the tokens:

  1. Thursday: 0.6 × 0.9 + 0.4 × 0.5 = 0.54 + 0.20 = 0.74.
  2. Zoom: 0.6 × 0.2 + 0.4 × 0.9 = 0.12 + 0.36 = 0.48.
  3. Product: 0.74 × 0.48 = 0.355.

On the slide this is: re-pick the passage at every token of the answer.

RAG-Token gets the higher probability, 0.355 against 0.288. It can lean on the queue log for Thursday and on the office hours page for Zoom. RAG-Sequence has to write the whole answer from a single passage, and no single passage knows both facts.

The two formulas hold the same pieces, with ∑ and in swapped places. ∏ means multiply everything over the range written below it, just as ∑ means add everything. Here yi is the i-th token of the answer, and y<i is all the tokens before it.

RAG-Sequence: p(y|x) ≈ ∑z p(z|x) ∏i p(yi | x, z, y<i)
RAG-Token: p(y|x) ≈ ∏i ∑z p(z|x) p(yi | x, z, y<i)

In RAG-Sequence the ∏ is inside: first the tokens are multiplied for each passage (0.18 and 0.45), then comes the ∑. In RAG-Token the ∑ is inside: first the passages are added for each token (0.74 and 0.48), then comes the ∏. In the results of step 5 the two methods come out almost equal.

Stanford slide: Deep Dive into RAG, the formula and RAG-Sequence and RAG-Token
Stanford's slide: Deep Dive into RAG. At the top are the retriever and the generator, in the middle the formula from step 2, and at the bottom the two lines on RAG-Sequence and RAG-Token. You can now read every line of it.

Think first

In the example above, set both passages' weights to 0.5. What probability do RAG-Sequence and RAG-Token give now?

Show the answer

RAG-Sequence: 0.5 × 0.18 + 0.5 × 0.45 = 0.315. RAG-Token: (0.5 × 0.9 + 0.5 × 0.5) × (0.5 × 0.2 + 0.5 × 0.9) = 0.70 × 0.55 = 0.385. RAG-Token is ahead again, because the two tokens still come from different passages.

4Training the retriever and the generator with one loss

RAG's training makes −ln of the right answer's probability small, and this single loss changes both the retriever and the generator. Training all the parts together with one loss is called end-to-end training. The data holds only question-answer pairs, yet this loss is how the retriever learns which passage deserves more weight.

The is the one from lesson 4: −ln of the right answer's probability p. In step 2, Thursday 9am got 0.70, so the loss is −ln 0.70 = 0.357. Over all training examples j, these losses add up:

∑j − log p(yj | xj)

The slide calls this the . It is “marginal” because p(y|x) itself comes from marginalization over passages, and the slide's log is this ln.

Which passage's score lowers the loss when it rises

In the table below, each row raises one score by 0.5 and leaves the other two alone. The weights here have 3 decimals, so the first row's loss is 0.355, not 0.357. my arithmetic

Score raised by 0.5Weights: queue log, office hours, forump(“Thu 9am” | x)Loss
None0.602, 0.299, 0.0990.7010.355
Queue log, 1.8 to 2.30.714, 0.215, 0.0720.7570.279
Office hours page, 1.1 to 1.60.504, 0.413, 0.0830.6680.403
Forum post, 0.0 to 0.50.565, 0.281, 0.1540.6640.409

When the queue log's score rose, the total probability went from 0.701 to 0.757, and the loss fell. When either of the other two rose, the total fell and the loss rose, even though the office hours page gives 0.5.

The reason lies in the weighted average. p(y|x) is a weighted average of the generator's numbers 0.9, 0.5 and 0.1, meaning each number times its weight, all added up. A weighted average always lies between the smallest and the largest number, and it moves toward whichever number gains weight.

Set the queue log aside for a moment. The office hours page and the forum post have weights in the ratio 3 : 1, that is 0.75 and 0.25 between them. Their average is 0.75 × 0.5 + 0.25 × 0.1 = 0.4. So the total is 0.6 × 0.9 + 0.4 × 0.4 = 0.54 + 0.16 = 0.70. Raising the queue log's score does not change that 3 : 1, because in softmax the other two exps stay the same. So if the queue log's weight is w, the total is w × 0.9 + (1 − w) × 0.4:

Queue log's weight wTotal probability
0.50.65
0.6, now0.70
0.70.75
0.80.80
1.00.90

For the office hours page the same arithmetic runs the other way, because its 0.5 is below 0.79, the average of the other two passages.

The rule: if the generator gives the right answer a higher probability with a passage than the current total, training raises that passage's score. If it gives a lower one, training lowers the score. Here the total is 0.70, so the queue log's score (0.9) goes up, and the scores of the office hours page (0.5) and the forum post (0.1) go down. this rule is not on the slides, it follows from the maths

Now run the same rule on the broken retriever. There the total is 0.27, so both the queue log (0.9) and the office hours page (0.5) go up, and the forum post (0.1) goes down. The queue log was never named in the data, yet training moves the retriever toward it. The last column of the widget in step 2 shows exactly this rule.

Which parts are trained

Training nudges every parameter a little in the direction that lowers the loss, and the gives that direction. Working backwards from the loss to get every parameter's gradient is called backpropagation, or backprop for short. In RAG the gradient reaches the generator's parameters θ and, inside the retriever, the query encoder.

The passage encoder and the index stay fixed. Changing the passage encoder would mean rebuilding the vectors of 21 million passages and then the index. from the paper So training improves things only from the question's side. It changes the query encoder so that q(x) gets larger dot products with the useful passages. If a passage's vector was built badly, training cannot fix it.

The dashed arrow at the top of the slide in step 1 says exactly this: End-to-End Backprop through q and pθ. The arrow reaches q and pθ, not the d(z) in the document index.

Think first

Press “All three equal” in the widget, so each passage has weight 1/3. What is the total probability of Thursday 9am? Which passage's score will training raise, which will it lower, and what happens to the office hours page?

Show the answer

The total is (0.9 + 0.5 + 0.1) ÷ 3 = 0.50. The queue log (0.9) is above it, so its score goes up. The forum post (0.1) is below, so its score goes down. The office hours page gives exactly 0.5, equal to the total, so its score stays where it is.

5Results on NQ: RAG, T5-11B and DPR

The paper measures RAG on question-answering datasets. Natural Questions (NQ) holds real questions people typed into Google search, with answers in Wikipedia. from the paper An answer counts as right only if it matches the correct answer exactly, and this measure is called Exact Match. The score is the percentage of such questions.

The table has two kinds of models. A closed-book model has no documents and answers from its parameters alone. An open-book model first searches documents, then answers.

ModelKindNQ
T5-11BClosed book, 11 billion parameters34.5
REALMOpen book, picks the answer out of a passage40.4
DPROpen book, picks the answer out of a passage41.5
RAG-TokenOpen book, writes the answer itself44.1
RAG-SequenceOpen book, writes the answer itself44.5

T5-11B is at 34.5 and RAG-Sequence at 44.5, which is 10 points ahead. Out of 1,000 questions, that is 345 against 445 answers exactly right. So RAG, with documents, gets 100 more questions in 1,000 exactly right than T5-11B with its parameters alone.

The DPR row means the full QA system built on top of the DPR retriever, which picks the answer out of a passage. RAG takes the same kind of retriever, writes the answer itself, and goes from 41.5 to 44.5. RAG-Token is only 0.4 behind RAG-Sequence.

RAG taught its retriever from question-answer pairs, but to do so it also changed the generator's parameters. Today you often get an LLM through a company's API, such as GPT or Gemini, and you cannot change its parameters. Lesson 6 starts from exactly this situation.

What the dense version adds

The dense version contains this whole lesson, followed by 5 more steps that take you to the level of the Stanford slides:

  • Step 6: the maths of the training rule. Rates, ∂, the weight after seeing the answer (the posterior), and the score's rate = prior − posterior.
  • Step 7: the generator's and the query encoder's rates from one loss, why the passage encoder stays fixed, and the paper's experiments on the retriever.
  • Step 8: RAG-Sequence and RAG-Token in depth: a posterior that changes at every token, Thorough and Fast decoding, and the effect of K.
  • Step 9: the paper's real scale and the full results table, with all four datasets.
  • Step 10: Stanford's six RAG slides, which you read yourself.

Open the dense version

3 points from this lesson

  1. RAG treats the passage as a hidden choice and takes a weighted sum over the top-K passages: p(y|x) ≈ ∑ pη(z|x) · pθ(y|x, z). So training needs no labels for the right passage.
  2. RAG-Sequence writes the whole answer with one passage and then adds, while RAG-Token adds at every token. When parts of the answer sit in different passages, RAG-Token gets the higher probability, as in 0.355 against 0.288.
  3. One loss trains both the query encoder and the generator, while the passage encoder and the index stay fixed. Training raises the score of any passage with which the right answer gets a higher probability than the total.

New terms in this lesson

RAG
Retrieval-Augmented Generation: joining a retriever and a generator into one model and training them together with one loss. For example, bringing the queue log and writing Thursday 9am from it.
Latent variable
A hidden choice that the data does not record. In RAG, the choice of which passage the answer came from. For example, a question-answer pair does not say that the answer came from the queue log.
η and θ
The names of the retriever's (eta) and the generator's (theta) parameters. That is why the retriever is written pη(z|x) and the generator pθ(y|x, z).
BERT
An encoder model Google released in 2018. Both of the RAG retriever's encoders, the query encoder and the passage encoder, are built from it.
MIPS
Maximum inner product search: finding the K passages in the index with the largest dot products. For example, the 5 highest scores among 21 million passages.
Generator
The model that reads the question and one passage and writes the answer, pθ(y|x, z). In RAG it is BART. For example, 0.9 for Thursday 9am with the queue log.
Encoder-decoder
The encoder turns the input into vectors, and the decoder looks at them while it writes one token at a time. BART is such a model.
Concatenate
Join two texts end to end into one text. For example, question: … followed by passage: ….
RAG-Sequence
The whole answer with each passage, then the weighted sum. For example, 0.6 × 0.18 + 0.4 × 0.45 = 0.288.
RAG-Token
A weighted sum at every token, then the product of the tokens. For example, 0.74 × 0.48 = 0.355.
End-to-end training
Training all the parts together with one loss. In RAG the query encoder and the generator both learn from the same −ln p(y|x). For example, −ln 0.70 = 0.357.
Backpropagation
Working backwards from the loss to get every parameter's gradient, backprop for short. In RAG it reaches the generator and the query encoder, but not the passage encoder.
Exact Match
An answer counts as right only if it matches the correct answer exactly. The score is the percentage of such questions. For example, RAG-Sequence's 44.5 on NQ.
Closed book and open book
Answering from parameters alone, or searching documents first and then answering. For example, on NQ the closed-book T5-11B scores 34.5 and the open-book RAG-Sequence 44.5.