Lesson 16 · Easy version

Agentic RAG: retrieval as a decision

In agentic RAG, retrieval is no longer a fixed step in a pipeline. At every step an LLM decides whether to retrieve, what to retrieve, and whether what came back is good enough. An LLM working this way is called an agent, and this lesson shows two ways of making these decisions: Self-RAG and CRAG.

The need for it showed up in lesson 11. For the question which office hour is least crowded, BM25 gave the old forum post 2.749 and the queue log 2.473. At K = 1 only the forum post reaches the prompt, and the LLM writes midweek afternoons [P4], where [P4] is that post's ID. This is the wrong answer from lesson 1, now with a citation. The opposite mistake happens too: even for what is 17 × 3? the pipeline searches 60,000 passages.

The ladder for this lesson

The ladder has 5 steps. Under each step you can see what you will be able to do after it.

  1. Agents: action, observation and the loopFollow an agent loop step by step, and count its scores by hand.
  2. Three decisions, and where a pipeline falls shortExplain each decision with one question, and read the agentic RAG slide.
  3. Self-RAG: decisions as tokensRank segments from their reflection tokens, and read the Self-RAG slide.
  4. CRAG: a separate evaluator and two thresholdsName CRAG's action from a confidence, and read the CRAG slide.
  5. Self-RAG and CRAG side by side, and their limitsSay who makes which decision, and why lesson 17 is needed.

What you need first: Lesson 1 (next-token prediction, vocabulary), Lesson 6 (pipeline, query rewriting, ITER-RETGEN) and Lesson 14 (reranker). If you already know a step, skip to the next one. A word with a dotted underline and ▾ opens its definition when you click it.

1Agents: action, observation and the loop

An agent is an LLM that works in a loop. It chooses the next thing to do, that thing gets done, and then it reads the result and chooses again. What the agent chooses to do is called an action, such as searching or writing the answer. A program actually carries out the action, a search engine for example, and that program is called a tool. What the tool returns, and what the agent reads before its next step, is called the observation.

The need for a loop showed up in lesson 6. There, searched twice, and a person had fixed that count in advance. But on the Jed Hoyer question the model made three searches, and 17 × 3 needs none. No fixed count suits both, so the count should come from whoever is reading the result of each search.

Here is a small loop. The user asks in shorthand, OH timings? The retriever is the word overlap from lesson 6, which counts how many query words also appear in a passage. my example loop

StepThe agent's actionObservation
1Search: OH timingsEvery passage scores 0
2Search: CS 329Z office hours scheduleOffice hours page 2, queue log 1
3Answer: Tuesday 4pm, Room 104, or Thursday 9am, Zoom [P2]Loop ends

The first query's words, oh and timings, appear in no passage. In the second query, office and hours both appear in the office hours page, so it scores 2. The queue log has office, but it says hour instead of hours, so it scores 1. The [P2] in the answer is the office hours page's ID.

The second step came from the first step's observation: nothing came back, so the agent rewrote the query. This is the from lesson 6. Lesson 6's Advanced RAG pipeline rewrites every query, which costs an extra LLM call on every query. Each step of the agent also costs an LLM call, but here the rewriting step happened only because the search came back empty.

Think first

Suppose the user had typed office hour timings in the first place. What would each passage score, and would the agent need to rewrite the query?

Show the answer

The queue log has both office and hour, so it scores 2. The office hours page has only office, so it scores 1. The search did not come back empty, so there is no reason to change the query. But the queue log has the times and not the place, so the agent now has to decide whether what it found is enough. That is the third decision in the next step.

2Three decisions, and where a pipeline falls short

At every step, an agent that retrieves faces three decisions. First, retrieve or answer directly. Second, which query to send. Third, whether what came back is good enough. All three appeared in the loop of step 1: the agent chose to search, then wrote a new query, then judged the result good enough and answered.

A pipeline makes none of these decisions. It runs the same steps on every question. Below is one question per decision where the pipeline falls short:

DecisionQuestionWhat the pipeline doesWhat an agent can do
1. Retrieve?What is 17 × 3?Searches 60,000 passagesand puts about 2,500 tokens into the prompt0 searches, answers 51 directly
2. Which query?OH timings?Sends that query as is, gets 0 passagesChanges the query when the search comes back empty
3. Good enough?Least crowded, K = 1Sends the forum post aloneThe post is a 2024 opinion, so it searches again

When retrieval becomes an action of the agent, and these three decisions are made at every step, the result is called agentic RAG. Stanford's slide says exactly this:

Stanford slide: agentic RAG, retrieval inside the loop, and three decisions
Stanford's slide: retrieval inside the loop. The top line says that nothing so far decides whether to retrieve, what to ask, or when to stop. Below are the same three decisions. Click the image to open it at full size.

The when to stop at the top and the third decision at the bottom are the same thing. If the result is good enough, the agent stops and answers, and if it is not, the agent searches again.

The question now is who makes these decisions. This lesson gives two answers. In Self-RAG the model that writes the answer makes them itself, and in CRAG a separate, small model checks the result.

3Self-RAG: decisions as tokens

The most direct option is to let the LLM that writes the answer make the decisions, because it is reading both the question and the passages. The difficulty is that an LLM only writes the . It does not announce decisions separately.

Self-RAG solves this by turning the decisions into tokens as well. It adds some special tokens to the model's , called reflection tokens. The model gets special training, and then it writes these tokens just like any other text. Self-RAG is a 2024 method by Asai and colleagues, and its main tokens are these:

TokenMeaningDecision
RetrieveSearch nowDecision 1
Relevant / IrrelevantWhether the retrieved passage is useful for the questionDecision 3
Supported / PartiallyWhether the passage fully or only partly backs up what the model wroteA second check for decision 3

The paper has a few more tokens, such as No support. A fourth kind gives the whole answer a score from 1 to 5, and it is covered in the dense version.

Now take our question: Which office hour this week is least crowded? Suppose the model wrote Retrieve, and the retriever brought the queue log, the old forum post and the Homework 1 announcement. The model writes the next segment separately with each passage, and adds reflection tokens to each segment. my example and tokens

PassageReflection tokensSegment the model wrote
Queue logRelevant, SupportedThursday 9am, about 3 waiting.
Old forum postRelevant, PartiallyWednesday afternoon is quietest.
Homework 1 announcement?Homework 1 will be released on Oct 5.

The forum post segment is Partially, because the post says midweek, not Wednesday, and it also says usually. So the Wednesday claim is only half in the passage. The queue log claim is written out in full in the queue log, so that segment is Supported.

Work out the third row

Which token goes on the Homework segment? What is the order of the three segments, and which one joins the answer?

Show the answer

Irrelevant. The Homework date is true, and the announcement even backs it up, but the question is about crowding at office hours. The order is the queue log, then the forum post, then Homework. The queue log segment is Relevant and Supported, so that is the one that joins the answer: Thursday 9am.

So Self-RAG checks each segment on two counts: whether it is useful for the question, and whether the passage backs it up. Stanford's slide shows the paper's own example, and you can now read it:

Stanford slide: the three steps of Self-RAG on the US states example
Stanford's slide: the three steps of Self-RAG. Green boxes are the good tokens, the orange box is Partially, and the red box is Irrelevant.

In step 1 the model writes a first sentence, then Retrieve, and three passages arrive. In step 2 it writes one segment with each passage, and in step 3 the tokens give the order 1 > 3 > 2. The Texas claim is true, but its passage is Irrelevant, so it comes last. Then the model writes Retrieve again and repeats the process.

Self-RAG has a cost. The generator needs special training, so a from an API does not work here. It has to write a separate segment with each passage, which means K generations, like the model of lesson 5. And Supported only says that the claim is in the passage. It does not say that the passage is still true.

4CRAG: a separate evaluator and two thresholds

Most systems are built on a frozen LLM, so decision 3 has to be made by something outside the generator. A first idea is the from lesson 14. But a reranker only changes the order. Suppose the retriever brought only the Homework 1 announcement and the project page, both useless for the crowding question. The reranker still puts one of them on top, and that one goes into the prompt. my example

In CRAG, a small model gives every document a confidence: how right this document is for this question. This model is called the retrieval evaluator. CRAG stands for Corrective RAG, a 2024 method by Yan and colleagues. The confidence has a meaning of its own, so you can set a limit on it in advance, and such a limit is called a threshold.

CRAG keeps two thresholds, an upper one and a lower one. Here the confidence runs from 0 to 1, and the thresholds are 0.6 and 0.2. my scale and numbers The paper's own numbers are in the dense version. Start with the case where the retriever brought a single document, K = 1:

Single documentConfidenceVerdict
Queue log0.85Correctabove 0.6
Old forum post0.4Ambiguousbetween 0.2 and 0.6
Homework 1 announcement0.05Incorrectbelow 0.2

With several documents, one document above 0.6 is enough for Correct, while Incorrect needs every document below 0.2. In the reranker example Homework is at 0.05 and the project page at 0.1, so the verdict is Incorrect. my 0.1 for the project page Each verdict triggers a different action:

VerdictActionWhat the generator gets
CorrectKnowledge refinement: split the documents into strips, drop the useless strips, join the restQuestion + cleaned documents
IncorrectDrop the documents, rewrite the query, search the webQuestion + web results
AmbiguousBothQuestion + cleaned documents + web results

CRAG's verdict at K = 1

The retriever brought one document. Change the evaluator's confidence and see what CRAG does. The thresholds are 0.6 and 0.2. my thresholds and confidences

0.85

Even on the Correct path the document does not go in whole. CRAG splits it into small pieces called strips. It then checks each strip with the evaluator, drops the useless strips, and joins the rest in order. This is called knowledge refinement. For the question When is the milestone due?, the syllabus's three sentences become three strips: my example

Syllabus stripKeep?
The project milestone is due Week 6, Thursday 11:59pm.Yes, this is the answer
Late days may not be applied to the milestone.Yes, it is about the milestone deadline
The final report is due Week 10.No, this is about the final report

The generator gets 2 of the 3 strips. The Week 10 line never reaches the prompt, so the generator cannot mistake it for the milestone date.

On the Incorrect path CRAG drops its documents. An LLM rewrites the question as keywords suited to a search engine, and CRAG searches the web. On the slide the question is Who was the screenwriter for Death of a Batman?, and the new query has three parts: Death of a Batman, screenwriter and Wikipedia.

Stanford slide: CRAG, the evaluator's three verdicts and their actions
Stanford's slide: CRAG. Correct runs Knowledge Refinement, Incorrect runs Knowledge Searching, and Ambiguous runs both. The bottom row shows what the generator gets for each verdict.

Watch out: Not every text from the web can be trusted. It can contain false claims, or even instructions that steer the agent off course, which come up in lesson 18. That is why the slide has a Select step after the web search too.

Think first

At K = 1 the forum post arrives alone, and the evaluator gives it 0.4. What does CRAG do, and what does the generator get?

Show the answer

0.4 is between 0.2 and 0.6, so the verdict is Ambiguous. CRAG refines the forum post and keeps it, and it also rewrites the query and searches the web. The generator gets the question, the cleaned part of the forum post and the web results.

5Self-RAG and CRAG side by side, and their limits

Now go back to the two questions from the start. On 17 × 3, a well-trained Self-RAG does not write Retrieve, and writes 51 directly. In the paper's own example, too, the model does not search for a prompt that asks for an essay. from the paper CRAG would search here as well, because it searches on every question and only checks the result.

If the forum post arrives alone at K = 1, Self-RAG's Wednesday segment is Partially, and the model can write Retrieve again. CRAG's evaluator gives the forum post 0.4, so CRAG also searches the web. Neither of them passes the forum post on unchecked any more.

Self-RAGCRAG
Who decidesThe generator itself, with reflection tokensA separate evaluator, with two thresholds
Does the generator change?Yes, special trainingNo, any LLM
Decision 1: retrieve?With the Retrieve tokenAlways retrieves
Decision 2: which query?Does not write a separate queryfrom the paperRewrites the query for the web
Decision 3: good enough?With the Relevant and Supported tokensWith the evaluator's confidence

Both have limits. A person sets CRAG's thresholds, and the evaluator can make mistakes too. Self-RAG's reflection tokens are also written by the model, so they can be wrong as well. And neither of them learned these decisions by trying things out.

In lesson 17 the model learns these decisions by trying and seeing the result: when to search, which query to write, and when to stop. In a loop, one wrong decision also affects the later steps, which is the subject of lesson 18.

What the dense version adds

The dense version contains this whole lesson, followed by 4 more steps that take you to the level of the Stanford slides:

  • Step 6: all four kinds of Self-RAG tokens, their probabilities, the retrieval threshold δ, and the weighted score of each segment.
  • Step 7: Self-RAG's training and the paper's results, read in points.
  • Step 8: CRAG's real evaluator, its −1 to 1 scale, the paper's thresholds, verdicts on sets of documents, and the results.
  • Step 9: Stanford's three agentic RAG slides, which you read yourself.

Open the dense version

3 points from this lesson

  1. An agent takes an action in a loop, reads the observation, and decides the next action. In agentic RAG, retrieval is one such action.
  2. Every step has three decisions: retrieve or not, which query to send, and whether the result is good enough. A pipeline makes none of them.
  3. Self-RAG makes these decisions inside the generator with reflection tokens. CRAG checks the result with a separate evaluator and two thresholds, and searches the web when retrieval goes wrong.

New terms in this lesson

Agent
An LLM that works in a loop: it chooses an action, reads the observation, and decides the next action. For example, when the search for OH timings came back empty, the agent wrote a new query.
Action and tool
What the agent chooses to do, such as searching for OH timings. A tool is the program that does it, such as a search engine.
Observation
The result of an action, which the agent reads before its next step. For example, the result of the first search: every passage scored 0.
Agentic RAG
RAG in which retrieval is an action of the agent, not a fixed step of a pipeline. For example, 0 searches for 17 × 3 and 3 for the Jed Hoyer question.
Self-RAG
The generator itself writes reflection tokens to decide when to retrieve and which segment to keep. For example, Relevant and Partially on the forum post segment.
Reflection tokens
Special tokens added to the vocabulary, which the model writes like any other text. For example, Retrieve, Relevant or Irrelevant, and Supported or Partially.
CRAG
Corrective RAG: an evaluator calls the retrieval Correct, Incorrect or Ambiguous, and refinement, web search or both follow. For example, the forum post at 0.4 is Ambiguous.
Retrieval evaluator
A small model that reads the question and one document and returns a confidence. For example, 0.4 for the forum post at K = 1.
Knowledge refinement
Splitting documents into strips, dropping the useless strips, and joining the rest in order. For example, 2 of the syllabus's 3 strips stay for the milestone question.
Threshold
A limit set in advance, where being above or below it triggers a different action. For example, in CRAG above 0.6 means Correct, and below 0.2 means Incorrect.