Lesson 6 · Dense version

RAG pipeline: Naive, Advanced and Modular

A RAG pipeline is a line of steps: from the query to retrieval, from retrieval to the prompt, and from the prompt to the LLM's answer. You cannot change the LLM, so to improve the answer you change the steps before it. Stanford's slide shows three forms of such a pipeline: Naive, Advanced and Modular RAG.

The need for this showed up at the end of lesson 5. There the retriever and the generator were trained together. But you cannot change the parameters of an LLM you use through an API, such as GPT or Gemini. Then both mistakes from lesson 2 come back. For when can I get help from a TA, the retriever returns no passage at all. For which office hour is least crowded, the old forum post gets the same score as the queue log.

The ladder for this lesson

The ladder has 10 steps. The first 5 are the same as in the easy version, and steps 6 to 10 take you to the level of the Stanford slides. Under each step you can see what you will be able to do after it.

  1. The frozen LLM and the pipelineExplain why lesson 5's training does not work with an LLM behind an API.
  2. Naive RAG: indexing time and query timeCount the cost of each query, and read the Naive RAG part of the slide.
  3. Pre-retrieval: improving the query before retrievalWork out the scores after query expansion by hand.
  4. Post-retrieval: improving the returned listExplain why reranking cannot bring back a missed passage, and read the Advanced RAG part of the slide.
  5. Modular RAG: modules and patternsWork through a search-twice pattern by hand, and read the Modular RAG part of the slide.
  6. From here: the Stanford level
  7. Naive and Advanced RAG in the surveyCompute precision, and prove with sets that expansion never lowers a score and reranking never adds a passage.
  8. All the modules of Modular RAGSay what each of the slide's ten modules does on our corpus.
  9. The DSP paper, and relative gainExplain how demonstrations get built automatically, and work out a relative gain.
  10. ITER-RETGEN: the formula, the results and multiplying chancesTie every symbol of the formula to your own numbers, and work out the chance that several searches all succeed.
  11. Read Stanford's slides yourselfExplain every line of this lesson's two Stanford slides.

What you need first: Lesson 2 (retriever, word overlap, stopwords, recall ceiling), Lesson 3 (embedding, index) and Lesson 5 (the RAG model, concatenate, Exact Match). Step 9 also uses Lesson 4 (conditional probability, chain rule). If you already know a step, skip to the next one. A word with a dotted underline and ▾ opens its definition when you click it.

1The frozen LLM: you can only change the prompt

A frozen LLM is an LLM whose parameters you cannot change. You can only send it a prompt and get an answer back. The of an LLM behind an API sit on the company's servers, so for you that LLM is frozen.

Lesson 5's RAG cannot be trained in this setting. There, used a single loss to change the parameters of both the retriever and the generator. A frozen LLM's parameters cannot change at all, so that route is closed.

What you can still change is everything before the LLM: the query, the retriever and the prompt. A pipeline is a line of steps in which the output of each step becomes the input of the next. For example, the retriever's output is the top-K passages, and those passages become the input of the prompt.

In a pipeline, the top-K passages all go into one prompt. In lesson 5 the generator ran separately with each passage, and the results were combined as a weighted sum. Here the LLM gets a prompt like this: my example

Answer using only the passages below. Cite passage IDs.
[P3] Queue log, last 3 weeks: Tuesday 4pm office hour averaged 14 students waiting; Thursday 9am office hour averaged 3.
[P2] Office hours: Tuesday 4pm in Room 104 and Thursday 9am on Zoom, run by the TAs.
Question: Which office hour is least crowded, and where is it?

The LLM itself picks which fact to take from which passage: Thursday 9am from the queue log, and Zoom from the office hours page. [P3] and [P2] are passage IDs from lesson 2's corpus, so the answer can cite them.

The price of this freedom is that the LLM learns nothing from its mistakes, and the retriever never even finds out about them. We are back to lesson 2, where the retriever and the reader were built separately. So every step has to be made good on its own, and the rest of this lesson shows those steps.

2Naive RAG: indexing time and query time

Naive RAG is the simplest pipeline: an index built in advance, and then retrieval, a prompt and the frozen LLM for every query. There is no extra step in between, which is why it is called naive, meaning plain. Its steps split into the same two parts that were in lesson 3.

In pipeline terms these two parts are called indexing time and query time. The indexing-time steps run once: split the documents into passages, make an of each passage, and store them all in an . An embedding is the vector an encoder makes from a piece of text.

Documents2,000 documents
Split into passages60,000 passages
Index60,000 embeddings, once
Indexing time: this line runs once. The numbers come from lesson 2's company: 2,000 documents × 30 paragraphs = 60,000 passages. my example

The query-time steps run for every query:

Query1 embedding
Retrieve60,000 dot products, top-5
Prompt5 × 500 tokens + the question
Frozen LLMRead, 1 call
Query time: this line runs for every query. my example

So every query costs 1 embedding, 60,000 and 1 LLM call. The prompt carries 5 × 500 = 2,500 tokens of passages. The two main query-time steps are called Retrieve and Read. In Read, the LLM reads the passages and writes the answer, like the in lesson 2.

The Naive RAG part of Stanford's slide
The left part of Stanford's slide: Naive RAG. From Documents to Indexing is indexing time, and from Query to Output is query time. The snowflake next to Frozen LLM marks it as frozen. Click the image to open it at full size.

Now run the two opening questions through this pipeline. For the retriever we use from lesson 2, because you can count every score by hand. Word overlap counts how many of the query's words also appear in a passage, and it first removes such as which and is.

QueryWhat the retriever returnedWhat the LLM got at K = 2
when can I get help from a TANothing. Every passage scores 0.Nothing. The office hours page, which has the answer, never reached the LLM.
which office hour is least crowdedQueue log (2), old forum post (2), office hours page (1)The queue log and the forum post: the real data plus an old opinion from 2024

The two mistakes are of different kinds. For the TA question the right passage never came back, and that is lesson 2's . For the least-crowded question the right passage came back, but a wrong passage came along with the same score, so this is a mistake in the order. Both mistakes happen before anything reaches the LLM, so no LLM, however good, can fix them.

Think first

500 new passages are added to the corpus, and one old document changes. Which Naive RAG steps have to run again, and which do not?

Show the answer

Only the indexing-time steps, and only for the text that changed. Make embeddings for the 500 new passages and add them to the index, and remake the embeddings of the changed document's passages. If you skip this, the index keeps returning the old text. The query-time steps run as before, and nothing changes in the LLM.

3Pre-retrieval: improving the query before retrieval

Pre-retrieval is the set of steps that improve the query before it goes to the retriever. The slide gives three of them: query routing, query rewriting and query expansion. Start with expansion on the TA question, because that question failed on the words of the query.

Without stopwords, this query keeps help and ta. The office hours page says TAs, not ta, and word overlap only matches exact words. Help appears in no passage. So the answer's passage is in the corpus, and only the query's words fail to match it.

Query expansion adds related words to the query, so that it can match the words in the passages. Add office hours to the TA query, and the retriever has four words: help, ta, office and hours. my example

PassageMatching wordsBefore expansionAfter expansion
Office hours pageoffice, hours02
Queue logoffice It says hour, not hours.01
The other four passagesnone00

Now the office hours page is on top and reaches the LLM. The added words usually come from an LLM given a short prompt: write search words related to this question.

The other two steps also work on the query. Query rewriting writes the whole query again. If a user types the short form OH timings?, an LLM turns it into CS 329Z office hours schedule. Query routing decides which index to search. For example, when is the project milestone due should go to the syllabus index.

Switch the pipeline's steps on and off

The retriever is lesson 2's word overlap. Expansion adds words to the query. Rerank sorts the returned passages again by a second score, and it comes in the next step. my rerank scores

2

#PassageOverlapRerank

These steps have a cost. Rewriting and expansion usually need one extra LLM call per query, so the answer is slower and more expensive. They can also change what the query means: if the model reads OH as Ohio, the search goes to the wrong place.

Think first

Pick which office hour is least crowded in the widget. Expansion adds waiting to this query. What are the new scores of the queue log and the old forum post? Then check in the widget.

Show the answer

The query's words are now office, hour, least, crowded and waiting. The queue log has office, hour and waiting, so it scores 3. The forum post has least and crowded, so it scores 2. The queue log is now alone at the top, but at K = 2 the forum post still goes into the prompt.

4Post-retrieval: improving the returned list

Post-retrieval is the set of steps that improve the retriever's list before it goes into the prompt. The slide gives three of them: rerank, summary and fusion. Start with reranking on the least-crowded question, because there the mistake was in the order.

The retriever gave the queue log and the forum post the same score, 2. It cannot see that the queue log is three weeks of data while the forum post is a single opinion from 2024.

The retriever is fast because it looks at every passage only roughly. The returned passages are few, so a slower but more accurate model can look at them. This model reads the query and the passage together, gives the passage a new score, and sorts the list again. This is called reranking, and the model is a reranker. How a reranker is built comes in lesson 14.

The retriever returned three passages, and the reranker puts the forum post at the bottom: my rerank scores

PassageRetriever scoreReranker scoreReaches the LLM at K = 2?
Queue log20.92Yes
Office hours page10.41Yes
Old forum post20.35No

At K = 2 the queue log and the office hours page now go through, and this pair gives the full answer. The queue log says Thursday 9am is the least crowded, and the office hours page says it is on Zoom. This is the prompt shown in step 1.

Think first

Pick the TA query in the widget. Leave Expansion off and turn Rerank on. Does the office hours page reach the LLM now?

Show the answer

No. The retriever returned no passage at all, so the reranker has nothing to sort. Now turn Expansion on as well: the office hours page enters the list, and reranking puts it on top.

A reranker only changes the order of the passages the retriever returned. So the two kinds of steps fix different mistakes. Pre-retrieval brings a missed passage into the list, and post-retrieval fixes the order of the list.

Two more steps come after retrieval. The first is summary: shortening the passages before they go into the prompt. Lesson 1's context window had room for 7,000 tokens of passages, which is 14 passages of 500 tokens. If each passage shrinks to 100 tokens, the same 14 passages take 14 × 100 = 1,400 tokens. my example The risk is that the summary drops a detail that matters, such as Room 104.

The second is fusion: merging several ranked lists into one. Suppose you searched with the original query and also with the rewritten query office hours schedule. The first list is queue log, forum post, office hours page. The second has only the office hours page (2) and the queue log (1). Those two passages appear in both lists, so they should rise to the top of the merged list. The exact way to merge comes in lesson 14.

The pipeline you get by adding pre-retrieval and post-retrieval steps to Naive RAG is called Advanced RAG.

The Advanced RAG part of Stanford's slide
The middle part of Stanford's slide: Advanced RAG. The orange boxes are the new steps: the three Pre-Retrieval steps above Retrieval, and the three Post-Retrieval steps below it.

5Modular RAG: modules and patterns

Modular RAG sees the pipeline as modules that you can connect in any order. A module is one part of the pipeline that does one job, such as Retrieve, Rerank or Read. A fixed order of modules is called a pattern. Naive RAG's pattern, for example, is Retrieve, then Read.

This matters for questions whose answer is spread over several passages. Advanced RAG still searches only once per question, so whatever the first search misses never arrives. A question that needs more than one search is called a multi-hop question.

Our corpus has such a question too: the one from the prompt in step 1, Which office hour is least crowded, and where is it? The first part of the answer is in the queue log, and the second part is in the office hours page. The office hours page gives the place next to Thursday 9am, so to find it you first need to know that the answer is Thursday 9am.

Now run the pattern Retrieve, Read, Retrieve, Read, in which the answer from the first Read is added to the query of the second search. K is 2, and the retriever is the same word overlap. my example

  1. First Retrieve: the query keeps office, hour, least, crowded and where, and where appears in no passage. The queue log and the forum post score 2 each, so those two come back.
  2. First Read: suppose the LLM goes with the queue log's data and writes: Thursday 9am. Neither passage says where it is.
  3. Second Retrieve: put the first answer, Thursday 9am, in front of the query. Now thursday and 9am count too.
  4. Second Read: at K = 2 the LLM gets the queue log and the office hours page, which is the prompt from step 1. It writes: Thursday 9am, on Zoom.
PassageFirst searchSecond search
Queue logoffice, hour, then also thursday, 9am24
Office hours pageoffice, then also thursday, 9am13
Old forum postleast, crowded22
Syllabusthursday, the second time01

This pattern is called ITER-RETGEN. Its paper also builds the next query by joining the previous answer to the question. from the paper Reranking only changes the order of the returned list, while ITER-RETGEN searches again with a new query.

It has a limit too. If the first Read gets the answer wrong, the second search heads the wrong way. If the LLM trusts the forum post and writes Wednesday, Thursday 9am never enters the second query.

The lecture also has a real multi-hop question: Who is older, Jed Hoyer or John William Henry II? The slide shows a model's trace on this question, which lesson 17 covers in full. The first search returns a passage about Jed Hoyer's career, with no date of birth in it. The next search is for John William Henry II, and its passage says: born September 13, 1949. The third search is for Jed Hoyer birth year, so each search depends on what the previous reading found.

Stanford slide: a model's trace on the Jed Hoyer question
Stanford's slide: the trace of a model called Search-R1. The <search> lines are the model's searches, and the <information> lines are the passages that came back.

Stanford's slide shows four patterns, taken from the survey by Gao and colleagues (2024):

PatternOrder of modulesWhat it means
Naive RAGRetrieve, ReadThe plain pipeline from step 2
Advanced RAGRewrite, Retrieve, Rerank, ReadWith the steps from steps 3 and 4 added
DSPDemonstrate, Search, PredictWorked examples, then searching, then the answer
ITER-RETGENRetrieve, Read, Retrieve, ReadSearching again with the first Read's answer

DSP's Demonstrate module puts a few worked examples in the prompt before the question, such as Q: When is the milestone due? A: Week 6, Thursday [P1]. my example Such examples are called demonstrations, and the method is called few-shot prompting. It differs from lesson 2's setup, where the prompt held only an instruction.

The modules part of the slide has 10 modules, and you have now seen 7 of them. The other three, Search, Predict and Memory, come in the dense version.

The modules in the Modular RAG part of Stanford's slide
The four patterns in the Modular RAG part of Stanford's slide
The right part of Stanford's slide: Modular RAG. The modules are at the top, and the four patterns below. In ITER-RETGEN, the arrow from the last Read back to the first Retrieve says that the sequence runs again.

A person writes the pattern in advance, so in ITER-RETGEN the number of searches is fixed, whether the question needs 1 search or 3. In lesson 16 and 17, the model itself decides when the loop stops and what the next query is. Every extra loop adds one more LLM call.

From here: the Stanford level

The 5 steps above are the same as in the easy version. The 5 steps below take you to the level of the Stanford slides: first Naive and Advanced RAG as the survey describes them, then all of the slide's modules, then the DSP and ITER-RETGEN papers, and finally reading both slides yourself. Every piece of maths is explained where it first appears.

6Naive and Advanced RAG in the survey: three problems, precision and sets

The diagram on Stanford's slide comes from a survey by Gao and colleagues. A survey is a paper that reads many papers in a field and arranges them into one framework. This one sorted more than 100 RAG papers into three forms: Naive, Advanced and Modular RAG. from the paper

Naive RAG's three stages and three problems

The survey describes Naive RAG in three stages: indexing, retrieval and generation. It also calls this the Retrieve-Read framework. In indexing, documents in formats such as HTML, Word and Markdown are first turned into plain text. The text is then split into chunks, and each chunk's embedding goes into a vector database. from the paper

In retrieval, the query's embedding is made by the same encoder that made the chunks' embeddings. Otherwise the query vector and the chunk vectors would be built in different ways, and their dot product would mean nothing. In generation, the query and the chosen chunks go into one prompt, just as in step 1.

The survey then lists three problems with Naive RAG. Each one comes with an example from our corpus below: my examples

ProblemWhat the survey saysIn our corpus
RetrievalLow precision and low recall: wrong chunks come back, and needed chunks are missed.At K = 2 the forum post came back, and for the TA question the office hours page was missed.
GenerationHallucination: the answer says something the returned text does not support.The LLM writes that the Thursday 9am office hour is in Room 104, while the passage says Zoom.
AugmentationCombining passages is hard: when several passages say the same thing, the answer repeats itself.The queue log and the office hours page both mention Tuesday 4pm and Thursday 9am.

The first row brings a new word, precision. Precision is the share of the returned passages that are right. For the least-crowded question at K = 2, the queue log is right and the forum post is wrong, so precision = 1 ÷ 2 = 0.5. Recall counts from the other side: how many of the needed passages came back. For the TA question one passage was needed and none came back, so recall = 0 ÷ 1 = 0.

The survey adds a fourth point: for a hard question, one search with the original query is often not enough. from the paper The multi-hop question in step 5 is exactly this problem.

In the language of sets: expansion never lowers a score, and reranking never adds a passage

Two claims from steps 3 and 4, the limits of expansion and of reranking, can be proved cleanly with sets. A is a group of things in which each thing appears once. It is written in curly brackets, so the set of words in the TA query is Q = {help, ta}. The number of things in a set is written with bars: |Q| = 2.

The things two sets have in common form their intersection, written ∩. The least-crowded query's set is {office, hour, least, crowded}. Its intersection with the queue log's words is {office, hour}, which holds 2 things. The word overlap score is exactly this count:

score(Q, P) = | Q ∩ P |

Here Q is the set of the query's words, and P is the set of the passage's words. For the queue log |{office, hour}| = 2, and for the forum post |{least, crowded}| = 2.

Putting two sets together into one is called their union, written ∪. That is what expansion does: it joins a set E of related words to the query's set Q. For the TA query, Q ∪ E = {help, ta} ∪ {office, hours} = {help, ta, office, hours}. For the office hours page, Q ∩ P was empty, and (Q ∪ E) ∩ P = {office, hours}.

Now the proof. A word in Q ∩ P is in Q and also in P. Every word of Q is also in Q ∪ E, so that word is in (Q ∪ E) ∩ P as well. So everything in the old intersection is still in the new one, and new things can join:

| (Q ∪ E) ∩ P | ≥ | Q ∩ P |

So expansion can never lower any passage's word overlap score. That is its strength, because a missed passage can rise. It is also its limit, because a wrong passage never drops either. After adding waiting, the forum post stayed at 2.

For reranking, call the set of returned passages R and the set of needed passages G. Reranking only changes the order of the things in R, so R stays the same, and so does |R ∩ G|. For the TA question R = { }, the empty set, and it is still empty after reranking. Reranking does not change recall. It can only change the precision of the top K. In step 4, at K = 2 the forum post was in the top two at first. After reranking, the office hours page took its place, and that page is needed for the where part.

The survey's other pre-retrieval and post-retrieval methods

On the slide, pre-retrieval holds only the three query steps. The survey also counts improving the index as pre-retrieval, and lists several forms of each step: from the paper

In the summary example, each passage went from 500 to 100 tokens, so it became 500 ÷ 100 = 5 times smaller. Then the same 7,000 tokens of room fit 7,000 ÷ 100 = 70 passages instead of 14. my arithmetic So a summary can either raise K, or make every query cheaper.

7All the modules of Modular RAG, and the survey's new patterns

The modules part of the slide has two rings. The inner ring, around the word RAG, holds Retrieve, Rewrite, Rerank and Read, the modules of the Advanced RAG pattern. The outer ring holds Search, Routing, Predict, Demonstrate, Memory and Fusion. The dashed lines say that modules can connect to each other in any order.

The Retrieve and Read boxes have orange borders, and the rest have blue ones. The patterns part uses the same colours, so orange probably marks Naive RAG's two original modules. my guess The table below shows what each module does on our corpus:

ModuleJobIn our corpus
RetrieveTop-K passages from the indexWord overlap, K = 2
ReadRead the passages and answerThursday 9am, on Zoom
RewriteWrite the query againOH timings? becomes CS 329Z office hours schedule
RerankSort the returned list againThe forum post at 0.35, at the bottom
RoutingPick the source or routeThe milestone question goes to the syllabus index
FusionMerge several listsThe queue log and the office hours page rise from two lists
DemonstrateWorked examples in the promptThe milestone Q and A
SearchSearch a search engine, database or knowledge graph directlyIf office hours lived in a database table, the LLM would write a query for it
PredictThe LLM writes the context itself instead of retrieving itThe LLM writes CS 329Z's office hours from memory
MemoryText the LLM wrote earlier guides the next retrievalThe first Read's Thursday 9am goes into the next search

The job column follows the survey. The corpus examples are mine.

According to the survey, the Search module also has the LLM write code or a query language, so that it can search sources such as databases. The Predict module aims to cut down repetition and noise in what retrieval returns. But our course's office hours were never in any LLM's training, so here Predict would make up a wrong answer. The survey also lists a Task Adapter module, which is not on the slide. It adapts RAG to a particular task. from the paper

In Modular RAG the order of the modules can change too. The survey lists some new patterns: from the paper

8DSP: passing text between a frozen LM and a retriever

DSP, short for Demonstrate–Search–Predict, is a program that solves hard questions by passing text back and forth between a frozen LM and a frozen retriever. Khattab and colleagues at Stanford built it in 2022. The paper uses GPT-3.5 as the LM, short for language model, and ColBERTv2 as the retriever. Neither is trained. from the paper

The paper's first example is this question: How many storeys are in the castle David Gregory inherited? It needs the castle's name first, and then its number of storeys. Three systems handle it differently: from the paper

DSP has three stages, and the slide has a module for each: from the paper

  1. Demonstrate: the program has a few question-answer pairs, such as 16, but no labels for the queries in between. It runs Search and Predict on these training questions itself. Whenever its answer matches the correct answer, the whole path, meaning the queries and passages, becomes a demonstration.
  2. Search: the LM writes the query for each hop, and the retriever brings its passages. On the multi-hop dataset HotPotQA the number of hops is fixed at 2.
  3. Predict: the LM reads the demonstrations and the passages and writes the final answer.

Here is the Demonstrate stage on our corpus. The training pair is the question when is the project milestone due, with the answer Week 6, Thursday 11:59pm. my example

  1. Search: the query keeps project, milestone and due. The syllabus scores 3, and the project page scores 2.
  2. Predict: reading the syllabus, the LM writes Week 6, Thursday 11:59pm. This matches the correct answer.
  3. So the whole path, the question, the syllabus and the answer, becomes a demonstration. Had the answer not matched, the program would have dropped this pair.

No person wrote the steps in between. The program chose them by checking against the correct answer. The paper calls this kind of loose labelling weak supervision. The demonstration in step 5, Week 6, Thursday [P1], could be built exactly this way.

DSP's results, and the maths of relative gain

The paper measures on 1,000 questions from the multi-hop dataset HotPotQA. The score is , the percentage of answers that match the correct answer exactly. In lesson 5, RAG-Sequence's Exact Match on NQ was 44.5. from the paper

SystemExact Match (%)
The LM alone, with 16 examples28.3
Self-ask, with an improved prompt28.6
Retrieve-then-read36.9
DSP51.4

The gap between DSP and retrieve-then-read can be written in two ways. The first is a plain subtraction: 51.4 − 36.9 = 14.5. Both numbers are percentages, so this gap is called 14.5 percentage points. The paper uses the second way, called relative gain: divide the gap by the old number.

  1. The gap: 51.4 − 36.9 = 14.5.
  2. Divide by the old number: 14.5 ÷ 36.9 = 0.393.
  3. Multiply by 100 to get a percentage: 0.393 × 100 = 39.3, about 39%. Check: 36.9 × 1.393 = 51.4.

The same arithmetic against the LM alone gives (51.4 − 28.3) ÷ 28.3 = 23.1 ÷ 28.3 = 0.816, about 82%. Against self-ask it gives (51.4 − 28.6) ÷ 28.6 = 0.797, about 80%. The paper reports exactly these three numbers: 82%, 39% and 80%.

A relative gain depends on the old number. Here is the same 14.5-point gap on different old numbers: my arithmetic

Old number14.5 ÷ old numberRelative gain
10.01.450145%
28.30.51251%
36.90.39339%
50.00.29029%

The smaller the number you divide by, the bigger the relative gain. So against a weak baseline a percentage can look very large. When you read a paper, always look at both actual numbers.

Think first

Another result from the paper: on Open-SQuAD, DSP scores 36.6 and retrieve-then-read scores 33.8. What is the gap in percentage points, and what is the relative gain?

Show the answer

The gap is 36.6 − 33.8 = 2.8 points. The relative gain is 2.8 ÷ 33.8 = 0.083, about 8%, and the paper also reports 8%. On Open-SQuAD questions one search is often enough, so the gap there is small. my reasoning

9ITER-RETGEN: the formula, the results, and the maths of several searches

ITER-RETGEN runs retrieval and generation in turns: each time, the LLM's answer joined to the question becomes the query for the next search. Shao and colleagues built it in 2023. You ran it by hand on our corpus in step 5. Now look at the same arithmetic in the paper's formula: from the paper

yt = M( yt | prompt( Dyt−1 ‖ q , q ) ),  ∀ 1 ≤ t ≤ T

The table below ties each symbol to the numbers from step 5:

SymbolMeaningIn step 5
qThe questionWhich office hour is least crowded, and where is it?
t and Tt is the number of the iteration, and T the total number of iterations. ∀ means “for each”, so the formula holds for each t = 1, 2, …, T.T = 2, so t = 1 and t = 2
ytThe LLM's answer in iteration t. y0 is empty, so the first query is just q.y1 = Thursday 9am, y2 = Thursday 9am, on Zoom
‖Joining two texts end to end, the from lesson 5y1 ‖ q = Thursday 9am which office hour …
DxThe top-k passages returned when text x is used as the queryDq = {queue log, forum post}, Dy1‖q = {queue log, office hours page}
prompt(D, q)The prompt built from the passages and the questionA prompt like the one in step 1
M( y | … )The LLM, which writes the answer y given the promptThe frozen LLM

Read the “|” in the formula as “given”, as in lesson 4's . There, p(full | Tuesday) = 0.80 meant the probability of being full given that it is Tuesday. Here, M(yt | prompt) means that the LLM writes the answer yt given this prompt.

So the formula's two iterations are exactly the four steps of step 5. At t = 1 the query is y0 ‖ q = q, Dq holds the queue log and the forum post, and M writes y1 = Thursday 9am. At t = 2 the query is y1 ‖ q, and D holds the queue log (4) and the office hours page (3). The final answer is yT = y2.

The paper's example and setup

The paper's example is: What is the height of the player who won the 2015 AFL Rising Star award? In the first iteration, the retriever brings the award's page, which names the winner, Jesse Hogan. The LLM writes that Jesse Hogan is 198 cm tall, which is not in the returned text. The second iteration's query contains the name Jesse Hogan, so his own page comes back: 1.95 m. The answer is corrected to 1.95 m. from the paper

The paper's setup is as follows. The LLM is text-davinci-003, and the retriever is Contriever, a dense retriever. Each query returns the top 5 paragraphs, and the prompt holds 3 worked examples. The paper measures the first 500 questions of most datasets, and all 125 of Bamboogle. These include multi-hop datasets such as HotPotQA. from the paper

Besides Exact Match, the paper reports a second score, Acc†. Exact Match counts an answer as right only if it matches exactly. For it, 1.95 m and 195 cm are different answers. my example

So the paper also asks an LLM to judge whether the answer contains the correct answer, and calls that score Acc†. The paper checked by hand 80 answers on which the two scores disagreed, and Acc† was right 97.5% of the time. from the paper

The results: the second iteration adds the most

On HotPotQAExact MatchAcc†
Self-Ask36.864.8
DSP43.860.8
ITER-RETGEN, T = 139.265.5
ITER-RETGEN, T = 244.171.2
ITER-RETGEN, T = 345.271.4
ITER-RETGEN, T = 445.873.4

From the paper. T = 1 means a single retrieval, much like Naive RAG. The paper measured up to T = 7.

Multiplying chances across several searches

A multi-hop question gives every hop a new chance to fail. Suppose each search brings the right passage with a chance of 0.9. Out of 100 questions, the first search is right on 90. Of those 90, the second is also right on 90%, which is 90 × 0.9 = 81 questions. my numbers

So the chance that both are right is 0.9 × 0.9 = 0.81. Lesson 4's says the same: p(A and B) = p(A) × p(B | A), as in 0.5 × 0.8 = 0.40 there. Here A is the first search being right, and B the second.

HopsArithmeticChance all are right
10.90.9
20.9 × 0.90.81
30.9 × 0.9 × 0.90.729

In ITER-RETGEN, p(B | A) depends on A even more. If the first answer is Wednesday, Thursday 9am never enters the second query, and the second search's chance drops to almost 0. So every extra hop eats part of the accuracy, and the first mistake carries forward. Lesson 18 calls this , and there 0.9 multiplied over 5 steps gives 0.59.

Three ways to search more than once

The survey lists three ways to go beyond a single retrieval: from the paper

The Search-R1 trace on the slide is adaptive. The model writes its query inside a <search> tag, and the search system adds the returned passages inside an <information> tag. The model thinks inside <think>, and stops by writing its answer inside <answer>. from the paper The slide's title says the model learns these decisions with , which comes in lesson 17.

The trace also has a flaw. The third search, Jed Hoyer birth year, returns the same Jed Hoyer document as the first search, and it has no date of birth. Even so, the next think says that Jed Hoyer was born on December 7, 1973. That date did not come from the returned text, and lesson 17 covers what follows from this. The paper counts this trace as a successful case, and notes that the first query was not very meaningful. from the paper

10Now read Stanford's slides yourself

Below are this lesson's two Stanford slides, unchanged. Under each slide are a few questions. Think of the answer first, then open it. If something stops you, go back to the step named in the answer. If a slide looks small on a phone, click the image to open it at full size.

Stanford slide: Different Variants of RAG, that is, Naive, Advanced and Modular RAG
The Naive RAG part of the same slide, enlarged
The Advanced RAG part of the same slide, enlarged
The modules of the same slide, enlarged
The patterns of the same slide, enlarged
Stanford's Different Variants of RAG slide, and below it its four parts, enlarged.

Read the variants slide

  1. In Advanced RAG, the Pre-Retrieval box sits on the Query line, not on the Documents line. What does that mean? What else does the survey count as pre-retrieval?

    Show the answer

    All three pre-retrieval steps on the slide work on the query: routing, rewriting and expansion. The index line is the same as in Naive RAG. The survey also counts improving the index as pre-retrieval, such as the chunk size and date metadata. (Steps 3 and 6)

  2. Which four modules sit in the inner ring, and which pattern do they match? How does the Predict module differ from Retrieve?

    Show the answer

    Retrieve, Rewrite, Rerank and Read, which is the Advanced RAG pattern. Retrieve brings passages from the index, while in Predict the LLM writes the context itself. No LLM knows our course's office hours, so here Predict would make up a wrong answer. (Step 7)

  3. DSP is labelled [Khattab et al., 2022] and ITER-RETGEN [Shao et al., 2023]. Which one has the higher Exact Match on HotPotQA, and how many turns of ITER-RETGEN's arrow helped most?

    Show the answer

    In the ITER-RETGEN paper, DSP's Exact Match is 43.8 and ITER-RETGEN's at T = 2 is 44.1, almost the same. The gap in Acc† is larger, 60.8 against 71.2. The second iteration adds the most, 5.7 points, and after that the gains are small. In DSP's own paper its Exact Match was 51.4, but there the LM and the retriever were different. (Steps 8 and 9)

Stanford slide: the Search-R1 trace on the Jed Hoyer question
The first part of the same trace: the first two searches, enlarged
The second part of the same trace: the third search and the answer, enlarged
Stanford's Search-R1 slide, and below it the trace in two parts, enlarged.

Read the Search-R1 slide

  1. Which tags in the trace does the model write, and which does the search system write? What is the Ground Truth line for?

    Show the answer

    The model writes <think>, <search> and <answer>, and <information> holds the passages the search system returned. Ground Truth is the correct answer, John William Henry II. You check the model's <answer> against it. (Step 9, and lesson 17)

  2. How many searches did the model make? Which search's result came back a second time, and does it contain Jed Hoyer's date of birth?

    Show the answer

    Three searches. The third search returned the same Jed Hoyer document as the first. It has no date of birth, yet the next think writes December 7, 1973. That date did not come from retrieval. (Steps 5 and 9)

  3. How does this trace differ from ITER-RETGEN?

    Show the answer

    In ITER-RETGEN the number of searches T is fixed in advance, and the query is always yt−1 ‖ q. Here the model itself decides how many searches to make and what each query is, so it is adaptive. The slide's title says the model learns these decisions with RL. (Step 9, and lessons 16 and 17)

3 points from this lesson

  1. An LLM behind an API is frozen, so lesson 5's training does not work. A RAG pipeline improves the steps before the LLM: the query, retrieval and the prompt.
  2. Naive RAG builds the index once, and runs Retrieve and Read for every query. Expansion never lowers a passage's score, so it can bring in a missed passage. Reranking does not change the set of returned passages, so it cannot raise recall.
  3. Modular RAG connects modules in different orders. ITER-RETGEN searches again with the previous answer, and on HotPotQA the second iteration lifts Acc† from 65.5 to 71.2. Every extra hop multiplies the chances, as in 0.9 × 0.9 = 0.81.
Map of the rest of the lecture
Part of the pipelineQuestionLessons
IndexingHow are documents cut into pieces, and how is their context kept?7, 8, 9
RetrievalHow is search fast, and how do you search by words or meaning?10, 11, 12
MeasuringHow good is a retriever?13
Post-retrievalHow do you fuse and rerank?14, 15
LoopWhen should the model search, for what, and when stop?16, 17
FailuresWhere does it still break after all this?18

New terms in this lesson

Frozen LLM
An LLM whose parameters you cannot change. You can only send it prompts. For example, GPT or Gemini used through an API.
Pipeline
A line of steps in which the output of each step is the input of the next. For example, Retrieve outputs the top-K passages, and those are the input of Read.
Naive RAG
The simplest pipeline: an index, then Retrieve and Read for every query, with no extra step in between.
Indexing time and query time
Steps that run once, and steps that run for every query. For example, the embeddings of 60,000 passages are made once, and the query's embedding every time.
Pre-retrieval
Steps that improve the query before retrieval: routing, rewriting and expansion.
Query routing
Deciding which index or source to search. For example, “when is the project milestone due” goes to the syllabus index.
Query rewriting
Having an LLM turn the query into a clean search query. For example, “OH timings?” becomes “CS 329Z office hours schedule”.
Query expansion
Adding related words or other forms to the query. For example, adding “office hours” to “help ta” raises the office hours page from 0 to 2.
Post-retrieval
Steps that improve the returned list after retrieval: rerank, summary and fusion.
Rerank and reranker
A slower but more accurate model that sorts the returned passages again. For example, it puts the forum post below the queue log when both tie at 2.
Summary
Shortening passages before the prompt. For example, if a 500-token passage becomes 100 tokens, 14 passages take 1,400 tokens instead of 7,000. The risk: a detail such as Room 104 can drop out.
Fusion
Merging several ranked lists into one. For example, one list is queue log, forum post, office hours page, and another is office hours page, queue log. The passages that appear in both rise to the top.
Advanced RAG
The pipeline you get by adding pre-retrieval and post-retrieval steps to Naive RAG. For example, Rewrite, Retrieve, Rerank, Read.
Modular RAG
Splitting the pipeline into modules and connecting them in different orders. For example, Retrieve, Read, Retrieve, Read.
Pattern
A fixed order of modules, such as Retrieve, Read.
Demonstrations (few-shot)
Worked examples placed in the prompt before the question. For example, “Q: When is the milestone due? A: Week 6, Thursday [P1]”.
Multi-hop question
A question that needs several searches, because the parts of the answer sit in different passages. For example, where the least crowded office hour is: first the queue log, then the office hours page.
Precision
The share of the returned passages that are right. For example, at K = 2 the queue log is right and the forum post is wrong, so precision = 1 ÷ 2 = 0.5.
Relative gain
The difference divided by the old number, written as a percentage. For example, (51.4 − 36.9) ÷ 36.9 = 0.393, about 39%. The plain difference, 14.5, is in percentage points.
DSP
Demonstrate–Search–Predict: a program that passes text between a frozen LM and a frozen retriever, and builds its own demonstrations. Built at Stanford by Khattab and colleagues in 2022. For example, Exact Match 51.4 on HotPotQA.
ITER-RETGEN
Retrieval and generation in turns, where the previous answer joined to the question becomes the next query. Built by Shao and colleagues in 2023. For example, adding Thursday 9am lifts the office hours page from 1 to 3.
Iterative, recursive and adaptive retrieval
Three ways to search more than once: in turns for a fixed count, by breaking the question into smaller ones, and by letting the model decide. For example, ITER-RETGEN is iterative, and the Search-R1 trace is adaptive.