Lesson 6 · Easy version

RAG pipeline: Naive, Advanced and Modular

A RAG pipeline is a line of steps: from the query to retrieval, from retrieval to the prompt, and from the prompt to the LLM's answer. You cannot change the LLM, so to improve the answer you change the steps before it. Stanford's slide shows three forms of such a pipeline: Naive, Advanced and Modular RAG.

The need for this showed up at the end of lesson 5. There the retriever and the generator were trained together. But you cannot change the parameters of an LLM you use through an API, such as GPT or Gemini. Then both mistakes from lesson 2 come back. For when can I get help from a TA, the retriever returns no passage at all. For which office hour is least crowded, the old forum post gets the same score as the queue log.

The ladder for this lesson

The ladder has 5 steps. Under each step you can see what you will be able to do after it.

  1. The frozen LLM and the pipelineExplain why lesson 5's training does not work with an LLM behind an API.
  2. Naive RAG: indexing time and query timeCount the cost of each query, and read the Naive RAG part of the slide.
  3. Pre-retrieval: improving the query before retrievalWork out the scores after query expansion by hand.
  4. Post-retrieval: improving the returned listExplain why reranking cannot bring back a missed passage, and read the Advanced RAG part of the slide.
  5. Modular RAG: modules and patternsWork through a search-twice pattern by hand, and read the Modular RAG part of the slide.

What you need first: Lesson 2 (retriever, word overlap, stopwords, recall ceiling), Lesson 3 (embedding, index) and Lesson 5 (the RAG model). If you already know a step, skip to the next one. A word with a dotted underline and ▾ opens its definition when you click it.

1The frozen LLM: you can only change the prompt

A frozen LLM is an LLM whose parameters you cannot change. You can only send it a prompt and get an answer back. The of an LLM behind an API sit on the company's servers, so for you that LLM is frozen.

Lesson 5's RAG cannot be trained in this setting. There, used a single loss to change the parameters of both the retriever and the generator. A frozen LLM's parameters cannot change at all, so that route is closed.

What you can still change is everything before the LLM: the query, the retriever and the prompt. A pipeline is a line of steps in which the output of each step becomes the input of the next. For example, the retriever's output is the top-K passages, and those passages become the input of the prompt.

In a pipeline, the top-K passages all go into one prompt. In lesson 5 the generator ran separately with each passage, and the results were combined as a weighted sum. Here the LLM gets a prompt like this: my example

Answer using only the passages below. Cite passage IDs.
[P3] Queue log, last 3 weeks: Tuesday 4pm office hour averaged 14 students waiting; Thursday 9am office hour averaged 3.
[P2] Office hours: Tuesday 4pm in Room 104 and Thursday 9am on Zoom, run by the TAs.
Question: Which office hour is least crowded, and where is it?

The LLM itself picks which fact to take from which passage: Thursday 9am from the queue log, and Zoom from the office hours page. [P3] and [P2] are passage IDs from lesson 2's corpus, so the answer can cite them.

The price of this freedom is that the LLM learns nothing from its mistakes, and the retriever never even finds out about them. We are back to lesson 2, where the retriever and the reader were built separately. So every step has to be made good on its own, and the rest of this lesson shows those steps.

2Naive RAG: indexing time and query time

Naive RAG is the simplest pipeline: an index built in advance, and then retrieval, a prompt and the frozen LLM for every query. There is no extra step in between, which is why it is called naive, meaning plain. Its steps split into the same two parts that were in lesson 3.

In pipeline terms these two parts are called indexing time and query time. The indexing-time steps run once: split the documents into passages, make an of each passage, and store them all in an . An embedding is the vector an encoder makes from a piece of text.

Documents2,000 documents
Split into passages60,000 passages
Index60,000 embeddings, once
Indexing time: this line runs once. The numbers come from lesson 2's company: 2,000 documents × 30 paragraphs = 60,000 passages. my example

The query-time steps run for every query:

Query1 embedding
Retrieve60,000 dot products, top-5
Prompt5 × 500 tokens + the question
Frozen LLMRead, 1 call
Query time: this line runs for every query. my example

So every query costs 1 embedding, 60,000 and 1 LLM call. The prompt carries 5 × 500 = 2,500 tokens of passages. The two main query-time steps are called Retrieve and Read. In Read, the LLM reads the passages and writes the answer, like the in lesson 2.

The Naive RAG part of Stanford's slide
The left part of Stanford's slide: Naive RAG. From Documents to Indexing is indexing time, and from Query to Output is query time. The snowflake next to Frozen LLM marks it as frozen. Click the image to open it at full size.

Now run the two opening questions through this pipeline. For the retriever we use from lesson 2, because you can count every score by hand. Word overlap counts how many of the query's words also appear in a passage, and it first removes such as which and is.

QueryWhat the retriever returnedWhat the LLM got at K = 2
when can I get help from a TANothing. Every passage scores 0.Nothing. The office hours page, which has the answer, never reached the LLM.
which office hour is least crowdedQueue log (2), old forum post (2), office hours page (1)The queue log and the forum post: the real data plus an old opinion from 2024

The two mistakes are of different kinds. For the TA question the right passage never came back, and that is lesson 2's . For the least-crowded question the right passage came back, but a wrong passage came along with the same score, so this is a mistake in the order. Both mistakes happen before anything reaches the LLM, so no LLM, however good, can fix them.

Think first

500 new passages are added to the corpus, and one old document changes. Which Naive RAG steps have to run again, and which do not?

Show the answer

Only the indexing-time steps, and only for the text that changed. Make embeddings for the 500 new passages and add them to the index, and remake the embeddings of the changed document's passages. If you skip this, the index keeps returning the old text. The query-time steps run as before, and nothing changes in the LLM.

3Pre-retrieval: improving the query before retrieval

Pre-retrieval is the set of steps that improve the query before it goes to the retriever. The slide gives three of them: query routing, query rewriting and query expansion. Start with expansion on the TA question, because that question failed on the words of the query.

Without stopwords, this query keeps help and ta. The office hours page says TAs, not ta, and word overlap only matches exact words. Help appears in no passage. So the answer's passage is in the corpus, and only the query's words fail to match it.

Query expansion adds related words to the query, so that it can match the words in the passages. Add office hours to the TA query, and the retriever has four words: help, ta, office and hours. my example

PassageMatching wordsBefore expansionAfter expansion
Office hours pageoffice, hours02
Queue logoffice It says hour, not hours.01
The other four passagesnone00

Now the office hours page is on top and reaches the LLM. The added words usually come from an LLM given a short prompt: write search words related to this question.

The other two steps also work on the query. Query rewriting writes the whole query again. If a user types the short form OH timings?, an LLM turns it into CS 329Z office hours schedule. Query routing decides which index to search. For example, when is the project milestone due should go to the syllabus index.

Switch the pipeline's steps on and off

The retriever is lesson 2's word overlap. Expansion adds words to the query. Rerank sorts the returned passages again by a second score, and it comes in the next step. my rerank scores

2

#PassageOverlapRerank

These steps have a cost. Rewriting and expansion usually need one extra LLM call per query, so the answer is slower and more expensive. They can also change what the query means: if the model reads OH as Ohio, the search goes to the wrong place.

Think first

Pick which office hour is least crowded in the widget. Expansion adds waiting to this query. What are the new scores of the queue log and the old forum post? Then check in the widget.

Show the answer

The query's words are now office, hour, least, crowded and waiting. The queue log has office, hour and waiting, so it scores 3. The forum post has least and crowded, so it scores 2. The queue log is now alone at the top, but at K = 2 the forum post still goes into the prompt.

4Post-retrieval: improving the returned list

Post-retrieval is the set of steps that improve the retriever's list before it goes into the prompt. The slide gives three of them: rerank, summary and fusion. Start with reranking on the least-crowded question, because there the mistake was in the order.

The retriever gave the queue log and the forum post the same score, 2. It cannot see that the queue log is three weeks of data while the forum post is a single opinion from 2024.

The retriever is fast because it looks at every passage only roughly. The returned passages are few, so a slower but more accurate model can look at them. This model reads the query and the passage together, gives the passage a new score, and sorts the list again. This is called reranking, and the model is a reranker. How a reranker is built comes in lesson 14.

The retriever returned three passages, and the reranker puts the forum post at the bottom: my rerank scores

PassageRetriever scoreReranker scoreReaches the LLM at K = 2?
Queue log20.92Yes
Office hours page10.41Yes
Old forum post20.35No

At K = 2 the queue log and the office hours page now go through, and this pair gives the full answer. The queue log says Thursday 9am is the least crowded, and the office hours page says it is on Zoom. This is the prompt shown in step 1.

Think first

Pick the TA query in the widget. Leave Expansion off and turn Rerank on. Does the office hours page reach the LLM now?

Show the answer

No. The retriever returned no passage at all, so the reranker has nothing to sort. Now turn Expansion on as well: the office hours page enters the list, and reranking puts it on top.

A reranker only changes the order of the passages the retriever returned. So the two kinds of steps fix different mistakes. Pre-retrieval brings a missed passage into the list, and post-retrieval fixes the order of the list.

Two more steps come after retrieval. The first is summary: shortening the passages before they go into the prompt. Lesson 1's context window had room for 7,000 tokens of passages, which is 14 passages of 500 tokens. If each passage shrinks to 100 tokens, the same 14 passages take 14 × 100 = 1,400 tokens. my example The risk is that the summary drops a detail that matters, such as Room 104.

The second is fusion: merging several ranked lists into one. Suppose you searched with the original query and also with the rewritten query office hours schedule. The first list is queue log, forum post, office hours page. The second has only the office hours page (2) and the queue log (1). Those two passages appear in both lists, so they should rise to the top of the merged list. The exact way to merge comes in lesson 14.

The pipeline you get by adding pre-retrieval and post-retrieval steps to Naive RAG is called Advanced RAG.

The Advanced RAG part of Stanford's slide
The middle part of Stanford's slide: Advanced RAG. The orange boxes are the new steps: the three Pre-Retrieval steps above Retrieval, and the three Post-Retrieval steps below it.

5Modular RAG: modules and patterns

Modular RAG sees the pipeline as modules that you can connect in any order. A module is one part of the pipeline that does one job, such as Retrieve, Rerank or Read. A fixed order of modules is called a pattern. Naive RAG's pattern, for example, is Retrieve, then Read.

This matters for questions whose answer is spread over several passages. Advanced RAG still searches only once per question, so whatever the first search misses never arrives. A question that needs more than one search is called a multi-hop question.

Our corpus has such a question too: the one from the prompt in step 1, Which office hour is least crowded, and where is it? The first part of the answer is in the queue log, and the second part is in the office hours page. The office hours page gives the place next to Thursday 9am, so to find it you first need to know that the answer is Thursday 9am.

Now run the pattern Retrieve, Read, Retrieve, Read, in which the answer from the first Read is added to the query of the second search. K is 2, and the retriever is the same word overlap. my example

  1. First Retrieve: the query keeps office, hour, least, crowded and where, and where appears in no passage. The queue log and the forum post score 2 each, so those two come back.
  2. First Read: suppose the LLM goes with the queue log's data and writes: Thursday 9am. Neither passage says where it is.
  3. Second Retrieve: put the first answer, Thursday 9am, in front of the query. Now thursday and 9am count too.
  4. Second Read: at K = 2 the LLM gets the queue log and the office hours page, which is the prompt from step 1. It writes: Thursday 9am, on Zoom.
PassageFirst searchSecond search
Queue logoffice, hour, then also thursday, 9am24
Office hours pageoffice, then also thursday, 9am13
Old forum postleast, crowded22
Syllabusthursday, the second time01

This pattern is called ITER-RETGEN. Its paper also builds the next query by joining the previous answer to the question. from the paper Reranking only changes the order of the returned list, while ITER-RETGEN searches again with a new query.

It has a limit too. If the first Read gets the answer wrong, the second search heads the wrong way. If the LLM trusts the forum post and writes Wednesday, Thursday 9am never enters the second query.

The lecture also has a real multi-hop question: Who is older, Jed Hoyer or John William Henry II? The slide shows a model's trace on this question, which lesson 17 covers in full. The first search returns a passage about Jed Hoyer's career, with no date of birth in it. The next search is for John William Henry II, and its passage says: born September 13, 1949. The third search is for Jed Hoyer birth year, so each search depends on what the previous reading found.

Stanford slide: a model's trace on the Jed Hoyer question
Stanford's slide: the trace of a model called Search-R1. The <search> lines are the model's searches, and the <information> lines are the passages that came back.

Stanford's slide shows four patterns, taken from the survey by Gao and colleagues (2024):

PatternOrder of modulesWhat it means
Naive RAGRetrieve, ReadThe plain pipeline from step 2
Advanced RAGRewrite, Retrieve, Rerank, ReadWith the steps from steps 3 and 4 added
DSPDemonstrate, Search, PredictWorked examples, then searching, then the answer
ITER-RETGENRetrieve, Read, Retrieve, ReadSearching again with the first Read's answer

DSP's Demonstrate module puts a few worked examples in the prompt before the question, such as Q: When is the milestone due? A: Week 6, Thursday [P1]. my example Such examples are called demonstrations, and the method is called few-shot prompting. It differs from lesson 2's setup, where the prompt held only an instruction.

The modules part of the slide has 10 modules, and you have now seen 7 of them. The other three, Search, Predict and Memory, come in the dense version.

The modules in the Modular RAG part of Stanford's slide
The four patterns in the Modular RAG part of Stanford's slide
The right part of Stanford's slide: Modular RAG. The modules are at the top, and the four patterns below. In ITER-RETGEN, the arrow from the last Read back to the first Retrieve says that the sequence runs again.

A person writes the pattern in advance, so in ITER-RETGEN the number of searches is fixed, whether the question needs 1 search or 3. In lesson 16 and 17, the model itself decides when the loop stops and what the next query is. Every extra loop adds one more LLM call.

What the dense version adds

The dense version contains this whole lesson, followed by 5 more steps that take you to the level of the Stanford slides:

  • Step 6: the survey's three problems with Naive RAG, precision, and the limits of expansion and reranking in the language of sets.
  • Step 7: the slide's remaining modules, such as Search, Predict and Memory, and the survey's new patterns.
  • Step 8: the DSP paper, and the maths of relative gain.
  • Step 9: ITER-RETGEN's formula and the paper's results, and how chances multiply across several searches.
  • Step 10: both of this lesson's Stanford slides, which you read yourself.

Open the dense version

3 points from this lesson

  1. An LLM behind an API is frozen, so lesson 5's training does not work. A RAG pipeline improves the steps before the LLM: the query, retrieval and the prompt.
  2. Naive RAG builds the index once, and runs Retrieve and Read for every query. Pre-retrieval improves the query and brings in a missed passage. Post-retrieval fixes the order of the list, but cannot bring in a missed passage.
  3. Modular RAG connects modules in different orders. For a multi-hop question, ITER-RETGEN searches again with the first Read's answer.

New terms in this lesson

Frozen LLM
An LLM whose parameters you cannot change. You can only send it prompts. For example, GPT or Gemini used through an API.
Pipeline
A line of steps in which the output of each step is the input of the next. For example, Retrieve outputs the top-K passages, and those are the input of Read.
Naive RAG
The simplest pipeline: an index, then Retrieve and Read for every query, with no extra step in between.
Indexing time and query time
Steps that run once, and steps that run for every query. For example, the embeddings of 60,000 passages are made once, and the query's embedding every time.
Pre-retrieval
Steps that improve the query before retrieval: routing, rewriting and expansion.
Query routing
Deciding which index or source to search. For example, “when is the project milestone due” goes to the syllabus index.
Query rewriting
Having an LLM turn the query into a clean search query. For example, “OH timings?” becomes “CS 329Z office hours schedule”.
Query expansion
Adding related words or other forms to the query. For example, adding “office hours” to “help ta” raises the office hours page from 0 to 2.
Post-retrieval
Steps that improve the returned list after retrieval: rerank, summary and fusion.
Rerank and reranker
A slower but more accurate model that sorts the returned passages again. For example, it puts the forum post below the queue log when both tie at 2.
Summary
Shortening passages before the prompt. For example, if a 500-token passage becomes 100 tokens, 14 passages take 1,400 tokens instead of 7,000. The risk: a detail such as Room 104 can drop out.
Fusion
Merging several ranked lists into one. For example, one list is queue log, forum post, office hours page, and another is office hours page, queue log. The passages that appear in both rise to the top.
Advanced RAG
The pipeline you get by adding pre-retrieval and post-retrieval steps to Naive RAG. For example, Rewrite, Retrieve, Rerank, Read.
Modular RAG
Splitting the pipeline into modules and connecting them in different orders. For example, Retrieve, Read, Retrieve, Read.
Pattern
A fixed order of modules, such as Retrieve, Read.
Demonstrations (few-shot)
Worked examples placed in the prompt before the question. For example, “Q: When is the milestone due? A: Week 6, Thursday [P1]”.
Multi-hop question
A question that needs several searches, because the parts of the answer sit in different passages. For example, where the least crowded office hour is: first the queue log, then the office hours page.