Lesson 16 · Dense version

Agentic RAG: retrieval as a decision

In agentic RAG, retrieval is no longer a fixed step in a pipeline. At every step an LLM decides whether to retrieve, what to retrieve, and whether what came back is good enough. An LLM working this way is called an agent, and this lesson shows two ways of making these decisions: Self-RAG and CRAG.

The need for it showed up in lesson 11. For the question which office hour is least crowded, BM25 gave the old forum post 2.749 and the queue log 2.473. At K = 1 only the forum post reaches the prompt, and the LLM writes midweek afternoons [P4], where [P4] is that post's ID. This is the wrong answer from lesson 1, now with a citation. The opposite mistake happens too: even for what is 17 × 3? the pipeline searches 60,000 passages.

The ladder for this lesson

The ladder has 9 steps. The first 5 are the same as in the easy version, and steps 6 to 9 take you to the level of the Stanford slides. Under each step you can see what you will be able to do after it.

  1. Agents: action, observation and the loopFollow an agent loop step by step, and count its scores by hand.
  2. Three decisions, and where a pipeline falls shortExplain each decision with one question, and read the agentic RAG slide.
  3. Self-RAG: decisions as tokensRank segments from their reflection tokens, and read the Self-RAG slide.
  4. CRAG: a separate evaluator and two thresholdsName CRAG's action from a confidence, and read the CRAG slide.
  5. Self-RAG and CRAG side by side, and their limitsSay who makes which decision, and why lesson 17 is needed.
  6. From here: the Stanford level
  7. The maths of Self-RAG: a segment's scoreTurn a token's probability into a share, decide retrieval from δ, and work out each segment's score by hand.
  8. Self-RAG's training, and the paper's resultsSay where the reflection tokens in the training data came from, and read the paper's tables in points.
  9. CRAG's real evaluator and thresholdsWork out the verdict on a set of documents on the paper's scale, and read the results in points and percent.
  10. Read Stanford's slides yourselfExplain every line and every box of the three agentic RAG slides.

What you need first: Lesson 1 (next-token prediction, vocabulary), Lesson 6 (pipeline, query rewriting, ITER-RETGEN) and Lesson 14 (reranker). From step 6 on, Lesson 4 (probability, softmax, ln, loss) and Lesson 13 (points) also help. If you already know a step, skip to the next one. A word with a dotted underline and ▾ opens its definition when you click it.

1Agents: action, observation and the loop

An agent is an LLM that works in a loop. It chooses the next thing to do, that thing gets done, and then it reads the result and chooses again. What the agent chooses to do is called an action, such as searching or writing the answer. A program actually carries out the action, a search engine for example, and that program is called a tool. What the tool returns, and what the agent reads before its next step, is called the observation.

The need for a loop showed up in lesson 6. There, searched twice, and a person had fixed that count in advance. But on the Jed Hoyer question the model made three searches, and 17 × 3 needs none. No fixed count suits both, so the count should come from whoever is reading the result of each search.

Here is a small loop. The user asks in shorthand, OH timings? The retriever is the word overlap from lesson 6, which counts how many query words also appear in a passage. my example loop

StepThe agent's actionObservation
1Search: OH timingsEvery passage scores 0
2Search: CS 329Z office hours scheduleOffice hours page 2, queue log 1
3Answer: Tuesday 4pm, Room 104, or Thursday 9am, Zoom [P2]Loop ends

The first query's words, oh and timings, appear in no passage. In the second query, office and hours both appear in the office hours page, so it scores 2. The queue log has office, but it says hour instead of hours, so it scores 1. The [P2] in the answer is the office hours page's ID.

The second step came from the first step's observation: nothing came back, so the agent rewrote the query. This is the from lesson 6. Lesson 6's Advanced RAG pipeline rewrites every query, which costs an extra LLM call on every query. Each step of the agent also costs an LLM call, but here the rewriting step happened only because the search came back empty.

Think first

Suppose the user had typed office hour timings in the first place. What would each passage score, and would the agent need to rewrite the query?

Show the answer

The queue log has both office and hour, so it scores 2. The office hours page has only office, so it scores 1. The search did not come back empty, so there is no reason to change the query. But the queue log has the times and not the place, so the agent now has to decide whether what it found is enough. That is the third decision in the next step.

2Three decisions, and where a pipeline falls short

At every step, an agent that retrieves faces three decisions. First, retrieve or answer directly. Second, which query to send. Third, whether what came back is good enough. All three appeared in the loop of step 1: the agent chose to search, then wrote a new query, then judged the result good enough and answered.

A pipeline makes none of these decisions. It runs the same steps on every question. Below is one question per decision where the pipeline falls short:

DecisionQuestionWhat the pipeline doesWhat an agent can do
1. Retrieve?What is 17 × 3?Searches 60,000 passagesand puts about 2,500 tokens into the prompt0 searches, answers 51 directly
2. Which query?OH timings?Sends that query as is, gets 0 passagesChanges the query when the search comes back empty
3. Good enough?Least crowded, K = 1Sends the forum post aloneThe post is a 2024 opinion, so it searches again

When retrieval becomes an action of the agent, and these three decisions are made at every step, the result is called agentic RAG. Stanford's slide says exactly this:

Stanford slide: agentic RAG, retrieval inside the loop, and three decisions
Stanford's slide: retrieval inside the loop. The top line says that nothing so far decides whether to retrieve, what to ask, or when to stop. Below are the same three decisions. Click the image to open it at full size.

The when to stop at the top and the third decision at the bottom are the same thing. If the result is good enough, the agent stops and answers, and if it is not, the agent searches again.

The question now is who makes these decisions. This lesson gives two answers. In Self-RAG the model that writes the answer makes them itself, and in CRAG a separate, small model checks the result.

3Self-RAG: decisions as tokens

The most direct option is to let the LLM that writes the answer make the decisions, because it is reading both the question and the passages. The difficulty is that an LLM only writes the . It does not announce decisions separately.

Self-RAG solves this by turning the decisions into tokens as well. It adds some special tokens to the model's , called reflection tokens. The model gets special training, and then it writes these tokens just like any other text. Self-RAG is a 2024 method by Asai and colleagues, and its main tokens are these:

TokenMeaningDecision
RetrieveSearch nowDecision 1
Relevant / IrrelevantWhether the retrieved passage is useful for the questionDecision 3
Supported / PartiallyWhether the passage fully or only partly backs up what the model wroteA second check for decision 3

The paper has a few more tokens, such as No support. A fourth kind gives the whole answer a score from 1 to 5, and it is covered in the dense version.

Now take our question: Which office hour this week is least crowded? Suppose the model wrote Retrieve, and the retriever brought the queue log, the old forum post and the Homework 1 announcement. The model writes the next segment separately with each passage, and adds reflection tokens to each segment. my example and tokens

PassageReflection tokensSegment the model wrote
Queue logRelevant, SupportedThursday 9am, about 3 waiting.
Old forum postRelevant, PartiallyWednesday afternoon is quietest.
Homework 1 announcement?Homework 1 will be released on Oct 5.

The forum post segment is Partially, because the post says midweek, not Wednesday, and it also says usually. So the Wednesday claim is only half in the passage. The queue log claim is written out in full in the queue log, so that segment is Supported.

Work out the third row

Which token goes on the Homework segment? What is the order of the three segments, and which one joins the answer?

Show the answer

Irrelevant. The Homework date is true, and the announcement even backs it up, but the question is about crowding at office hours. The order is the queue log, then the forum post, then Homework. The queue log segment is Relevant and Supported, so that is the one that joins the answer: Thursday 9am.

So Self-RAG checks each segment on two counts: whether it is useful for the question, and whether the passage backs it up. Stanford's slide shows the paper's own example, and you can now read it:

Stanford slide: the three steps of Self-RAG on the US states example
Stanford's slide: the three steps of Self-RAG. Green boxes are the good tokens, the orange box is Partially, and the red box is Irrelevant.

In step 1 the model writes a first sentence, then Retrieve, and three passages arrive. In step 2 it writes one segment with each passage, and in step 3 the tokens give the order 1 > 3 > 2. The Texas claim is true, but its passage is Irrelevant, so it comes last. Then the model writes Retrieve again and repeats the process.

Self-RAG has a cost. The generator needs special training, so a from an API does not work here. It has to write a separate segment with each passage, which means K generations, like the model of lesson 5. And Supported only says that the claim is in the passage. It does not say that the passage is still true.

4CRAG: a separate evaluator and two thresholds

Most systems are built on a frozen LLM, so decision 3 has to be made by something outside the generator. A first idea is the from lesson 14. But a reranker only changes the order. Suppose the retriever brought only the Homework 1 announcement and the project page, both useless for the crowding question. The reranker still puts one of them on top, and that one goes into the prompt. my example

In CRAG, a small model gives every document a confidence: how right this document is for this question. This model is called the retrieval evaluator. CRAG stands for Corrective RAG, a 2024 method by Yan and colleagues. The confidence has a meaning of its own, so you can set a limit on it in advance, and such a limit is called a threshold.

CRAG keeps two thresholds, an upper one and a lower one. Here the confidence runs from 0 to 1, and the thresholds are 0.6 and 0.2. my scale and numbers The paper's own numbers are in the dense version. Start with the case where the retriever brought a single document, K = 1:

Single documentConfidenceVerdict
Queue log0.85Correctabove 0.6
Old forum post0.4Ambiguousbetween 0.2 and 0.6
Homework 1 announcement0.05Incorrectbelow 0.2

With several documents, one document above 0.6 is enough for Correct, while Incorrect needs every document below 0.2. In the reranker example Homework is at 0.05 and the project page at 0.1, so the verdict is Incorrect. my 0.1 for the project page Each verdict triggers a different action:

VerdictActionWhat the generator gets
CorrectKnowledge refinement: split the documents into strips, drop the useless strips, join the restQuestion + cleaned documents
IncorrectDrop the documents, rewrite the query, search the webQuestion + web results
AmbiguousBothQuestion + cleaned documents + web results

CRAG's verdict at K = 1

The retriever brought one document. Change the evaluator's confidence and see what CRAG does. The thresholds are 0.6 and 0.2. my thresholds and confidences

0.85

Even on the Correct path the document does not go in whole. CRAG splits it into small pieces called strips. It then checks each strip with the evaluator, drops the useless strips, and joins the rest in order. This is called knowledge refinement. For the question When is the milestone due?, the syllabus's three sentences become three strips: my example

Syllabus stripKeep?
The project milestone is due Week 6, Thursday 11:59pm.Yes, this is the answer
Late days may not be applied to the milestone.Yes, it is about the milestone deadline
The final report is due Week 10.No, this is about the final report

The generator gets 2 of the 3 strips. The Week 10 line never reaches the prompt, so the generator cannot mistake it for the milestone date.

On the Incorrect path CRAG drops its documents. An LLM rewrites the question as keywords suited to a search engine, and CRAG searches the web. On the slide the question is Who was the screenwriter for Death of a Batman?, and the new query has three parts: Death of a Batman, screenwriter and Wikipedia.

Stanford slide: CRAG, the evaluator's three verdicts and their actions
Stanford's slide: CRAG. Correct runs Knowledge Refinement, Incorrect runs Knowledge Searching, and Ambiguous runs both. The bottom row shows what the generator gets for each verdict.

Watch out: Not every text from the web can be trusted. It can contain false claims, or even instructions that steer the agent off course, which come up in lesson 18. That is why the slide has a Select step after the web search too.

Think first

At K = 1 the forum post arrives alone, and the evaluator gives it 0.4. What does CRAG do, and what does the generator get?

Show the answer

0.4 is between 0.2 and 0.6, so the verdict is Ambiguous. CRAG refines the forum post and keeps it, and it also rewrites the query and searches the web. The generator gets the question, the cleaned part of the forum post and the web results.

5Self-RAG and CRAG side by side, and their limits

Now go back to the two questions from the start. On 17 × 3, a well-trained Self-RAG does not write Retrieve, and writes 51 directly. In the paper's own example, too, the model does not search for a prompt that asks for an essay. from the paper CRAG would search here as well, because it searches on every question and only checks the result.

If the forum post arrives alone at K = 1, Self-RAG's Wednesday segment is Partially, and the model can write Retrieve again. CRAG's evaluator gives the forum post 0.4, so CRAG also searches the web. Neither of them passes the forum post on unchecked any more.

Self-RAGCRAG
Who decidesThe generator itself, with reflection tokensA separate evaluator, with two thresholds
Does the generator change?Yes, special trainingNo, any LLM
Decision 1: retrieve?With the Retrieve tokenAlways retrieves
Decision 2: which query?Does not write a separate queryfrom the paperRewrites the query for the web
Decision 3: good enough?With the Relevant and Supported tokensWith the evaluator's confidence

Both have limits. A person sets CRAG's thresholds, and the evaluator can make mistakes too. Self-RAG's reflection tokens are also written by the model, so they can be wrong as well. And neither of them learned these decisions by trying things out.

In lesson 17 the model learns these decisions by trying and seeing the result: when to search, which query to write, and when to stop. In a loop, one wrong decision also affects the later steps, which is the subject of lesson 18.

From here: the Stanford level

The 5 steps above are the same as in the easy version. The 4 steps below take you to the level of the Stanford slides: first the maths of Self-RAG's tokens, then its training and results, then CRAG's real evaluator and its results. Finally you read the slides yourself. Every piece of maths is explained where it first appears.

6The maths of Self-RAG: from token probabilities to a segment score

Step 3 had three kinds of tokens, and I gave the order of the segments by looking at the tokens. The paper has four kinds of tokens, and the order comes from a number. Each kind has a few values of its own: from the paper

KindValuesWhat it checks
Retrieveyes, no, continueSearch now, do not search, or keep using the passage retrieved earlier
IsRelrelevant, irrelevantWhether the passage is useful for the question
IsSupfully supported, partially supported, no supportHow far the passage backs up the segment
IsUse5, 4, 3, 2, 1How useful the answer is for the question

The paper calls the last three kinds critique tokens, because they judge what was written. In each kind one value is the best, shown in bold in the table. The Supported of step 3 is fully supported here, and Partially is partially supported.

From a token's probability to a share

At every step the model gives each token in the whole vocabulary a , and together they add up to 1. Self-RAG picks out the tokens of one kind and shares their probabilities among themselves. Suppose that after the queue log segment, relevant has probability 0.54, irrelevant 0.06, and all other tokens together 0.40. my numbers

  1. Add the two IsRel tokens: 0.54 + 0.06 = 0.60.
  2. Divide relevant by that sum: 0.54 ÷ 0.60 = 0.9.

This 0.9 is relevant's share among the IsRel tokens only. The other 0.40 plays no part, so the share always stays between 0 and 1. The paper writes it like this:

sIsRel = p(relevant) ÷ ( p(relevant) + p(irrelevant) )

Here p(relevant) is the probability of the relevant token, 0.54, and the bottom is the sum of both, 0.60. In lesson 4's the share was found the same way: divide one number by the sum of its group.

The retrieval decision: the threshold δ

Decision 1 uses the same kind of share. Self-RAG retrieves when yes's share, between yes and no, is above a threshold. The paper writes this threshold as δ, read "delta":

retrieve if p(yes) ÷ ( p(yes) + p(no) ) > δ

Look at two questions, where p(yes) and p(no) are the probabilities of the Retrieve tokens: my numbers

Questionp(yes), p(no)yes's share
What is 17 × 3?0.03, 0.870.03 ÷ 0.90 = 0.033
The least-crowded question0.56, 0.240.56 ÷ 0.80 = 0.70

Now change δ and see which question retrieves:

δ17 × 3 (0.033)Least crowded (0.70)
0RetrievesRetrieves
0.2Does not retrieveRetrieves
0.5Does not retrieveRetrieves
0.8Does not retrieveDoes not retrieve

So the larger δ is, the less the model retrieves. The paper sets δ = 0.2 for most tasks, to make the model retrieve often. On a task called ALCE, where every claim needs a citation, it sets δ = 0. from the paper

Each segment's score: a weighted sum

Now the shares of the three critique kinds combine into one number. First each kind's share, by the paper's rules: from the paper

The IsUse sum is a kind of average. Suppose 5 has share 0.6, 4 has 0.3 and 3 has 0.1. Then sIsUse = 1 × 0.6 + 0.5 × 0.3 + 0 × 0.1 = 0.75. If all the share were on 5 it would be 1, and if all of it were on 1 it would be −1. So sIsUse stays between −1 and 1.

For our three segments, the model gave these shares. Within each group the shares add up to 1: my numbers

Segment fromrelevantfully, partially, no5, 4, 3, 2, 1
Queue log0.90.8, 0.15, 0.050.6, 0.3, 0.1, 0, 0
Old forum post0.70.2, 0.7, 0.10.2, 0.5, 0.3, 0, 0
Homework 1 announcement0.050.6, 0.3, 0.10, 0, 0.1, 0.3, 0.6

The queue log segment's score takes four steps:

  1. sIsRel = 0.9.
  2. sIsSup = 0.8 + 0.5 × 0.15 = 0.875.
  3. sIsUse = 1 × 0.6 + 0.5 × 0.3 + 0 × 0.1 = 0.75.
  4. S = 1.0 × 0.9 + 1.0 × 0.875 + 0.5 × 0.75 = 0.9 + 0.875 + 0.375 = 2.15.

In step 4 the three shares were added with weights. The paper's default weights are 1.0 for IsRel and IsSup, and 0.5 for IsUse. from the paper The formula is exactly these four steps:

S = wIsRel sIsRel + wIsSup sIsSup + wIsUse sIsUse

Here each w is a weight (1.0, 1.0 and 0.5), and each s is that kind's share (0.9, 0.875 and 0.75). The same arithmetic on all three segments:

SegmentsIsRel, sIsSup, sIsUseS
Queue log0.9, 0.875, 0.752.15
Old forum post0.7, 0.55, 0.451.475
Homework 1 announcement0.05, 0.75, −0.750.425

The order is the same as in step 3: the queue log, then the forum post, then Homework. Homework's sIsSup is 0.75, so its claim is well backed by the announcement. But sIsRel of 0.05 and sIsUse of −0.75 push it to the bottom.

Think first

Suppose only IsSup counted, so wIsRel = 0, wIsUse = 0 and wIsSup = 1. What would the order of the three segments be?

Show the answer

Then S is just sIsSup: queue log 0.875, Homework 0.75 and forum post 0.55. Homework moves above the forum post, because its claim is backed by its own passage, even though it has nothing to do with the question. That is why IsRel is needed: Supported only says that the claim is in the passage.

The paper's full score also adds the segment's own probability to S:

f(yt, d) = p(yt | x, d, y<t) + S

Here x is the question, d the passage, yt the t-th segment, and y<t all the segments written before it. The paper treats each segment as one sentence. p(yt | x, d, y<t) says how likely the model finds this sentence, given the question, the passage and the earlier segments. from the paper

Changing the weights changes the model's behaviour without any new training. Suppose wIsSup = 2. Then the scores are 3.025, 2.025 and 1.175, so the order stays the same, but being backed up counts for more. In the paper, raising the IsSup weight on the ASQA task raised citation precision, meaning more citations really backed up their claims. At the same time a fluency score called MAUVE went down. from the paper, trend read from a graph

Beam search: two unfinished answers at once

The paper does not move forward with just the single best segment at each step. It keeps the B unfinished answers with the highest scores, and extends each of them to the next step. This is called beam search, and B is the width of the beam. In the paper B = 2. from the paper

On our three segments, B = 2 means that the unfinished answers with the queue log segment (2.15) and the forum post segment (1.475) both go on to the next step. The Homework one (0.425) is dropped there. At the end the best complete answer is returned. With B = 1 this would be like the of lesson 1, only with a whole sentence instead of a token.

7Self-RAG's training, and the paper's results

A model can write reflection tokens only if the training data has these tokens in the right places. Adding tokens to every sentence by hand is expensive, so the paper has another model do it. A model that grades what something else wrote is called a critic. Self-RAG's training takes three steps: from the paper

  1. GPT-4 was given an instruction and a few examples for each kind, and asked to write the tokens. This gave 4,000 to 20,000 examples per kind, such as 19,317 for IsRel, 11,181 for IsSup and 3,831 for IsUse.
  2. A critic was trained from Llama 2 7B on these examples, so that from then on the critic writes tokens instead of GPT-4.
  3. The critic added reflection tokens and retrieved passages to 145,619 training examples. Then the generator, Llama 2 7B or 13B, was trained on these examples.

The critic exists only to build training data. Once trained, the generator writes the tokens itself, so no critic is needed when answering.

The generator's training uses the same as lesson 4: −ln of the probability of each next token, with reflection tokens included. For example, if a correct token's probability goes from 0.5 to 0.9, its −ln falls from 0.693 to 0.105. The passage text gets no loss, because the model did not write it. The retriever supplied it. The paper writes the same thing like this:

maxM 𝔼 [ log pM(y, r | x) ]

Here M is the generator, x the question, y the answer, and r its reflection tokens. 𝔼 means the average over all training examples. Making log as large as possible and making −log, the loss, as small as possible are the same thing.

The paper measured how trustworthy the critic is by comparing it with GPT-4's tokens: from the paper

KindAgreement of critic with GPT-4 (%)
Retrieve93.8
IsSup93.5
IsRel80.2
IsUse73.5

IsUse is the weakest, because the critic often confuses 5 and 4, and people disagree on these two as well. Human annotators also checked 50 PopQA answers. The model's IsRel tokens matched their judgement 95.0% of the time, and its IsSup tokens 90.0% of the time.

When answering, the paper took the top 5 passages from a dense retriever called Contriever. The weights were 1.0, 1.0 and 0.5, δ = 0.2, and the beam width 2. from the paper

The results, read in points

Lesson 13 introduced : the plain difference between two percentages. For example, 50.8% to 54.9% is 4.1 points. The paper's four tasks are these. PopQA has 1,399 questions about rare people and things, PubHealth has health claims to judge true or false, ARC-Challenge has multiple-choice science questions, and Bio asks for biographies. from the paper

TaskSelf-RAG 7BChatGPT + retrievalChatGPT
PopQAaccuracy54.950.829.3
PubHealthaccuracy72.454.770.1
ARC-Challengeaccuracy67.375.375.3
BioFactScore, the % of small claims that are true81.2–71.8

The paper also measured what happens when one part is removed, which is called an . This was done on a 7B model trained on 50,000 examples, which is why the first row is lower than in the table above: from the paper

What changedPopQAPubHealth
Full Self-RAG45.573.5
Never retrieves when answering24.773.0
Always uses only the top 1 passage41.873.1
Instead of δ, retrieves only when the model itself writes Retrieve = yes28.372.6
No critic tokens in the training data42.672.0

On PopQA, never retrieving drops accuracy from 45.5 to 24.7, a fall of 20.8 points. On PubHealth the same fall is only 0.5 points. So retrieval is not equally useful on every task, and that is the reason for decision 1. On PopQA the δ = 0.2 rule is 45.5 − 28.3 = 17.2 points better than retrieving only on Retrieve = yes, because it retrieves more often.

8CRAG's real evaluator, thresholds and results

The first figure of the CRAG paper shows the problem clearly. The question is Who was the screenwriter for Death of a Batman?, and the retriever brings a document about Batman (1989 film). The name Hamm appears in it, and the generator answers Hamm, which is wrong. from the paper

CRAG's evaluator is a model called T5-large, with 0.77 billion parameters. Self-RAG's critic is Llama 2 7B, about 9 times larger. The evaluator reads the question and one document together and returns a score between −1 and 1. In training, a correct document had the label 1, and a wrong one −1. from the paper

PopQA has about 14,000 questions, and the paper trained the evaluator on all of them except the 1,399 test questions. The same evaluator was then used on the other three tasks as well. Each question usually brings 10 documents, and the evaluator scores each one separately. from the paper

The −1 to 1 scale

In the easy version I put the confidence on a 0 to 1 scale. The paper's scale runs from −1 to 1, and one simple rule links the two: c = (s + 1) ÷ 2. Here s is the paper's score and c the 0 to 1 confidence. For example, s = −1 gives c = 0, s = 0 gives c = 0.5, and s = 1 gives c = 1. The rule adds 1 to every score and divides by 2, so the order is the same on both scales.

The paper set the thresholds by trying values on each dataset, so they are : from the paper, last column my arithmetic

DatasetUpperLowerOn 0 to 1
PopQA0.59−0.990.795, 0.005
PubHealth, ARC0.5−0.910.75, 0.045
Biography0.95−0.910.975, 0.045

The last column comes from c = (s + 1) ÷ 2. For PubHealth's lower threshold: (−0.91 + 1) ÷ 2 = 0.09 ÷ 2 = 0.045. The lower thresholds sit almost at −1, so Incorrect happens only when every document is almost completely useless.

Think first

In the easy version, the Homework 1 announcement alone at 0.05 was Incorrect. With PubHealth's thresholds, on the 0 to 1 scale, what is its verdict?

Show the answer

PubHealth's thresholds on the 0 to 1 scale are 0.75 and 0.045. 0.05 is neither above 0.75 nor below 0.045, so the verdict is Ambiguous. With the paper's thresholds, CRAG rarely throws documents away. It keeps them and also searches the web.

Verdicts on a set of documents

The paper's rule applies to the whole set: from the paper

Now take my scores on the paper's scale: queue log 0.70, forum post −0.20, Homework −0.95 and project page −0.97. my scores

What the retriever broughtPopQA0.59, −0.99PubHealth0.5, −0.91Biography0.95, −0.91
Queue log, forum post, HomeworkCorrectCorrectAmbiguous
Forum post aloneAmbiguousAmbiguousAmbiguous
Homework, project pageAmbiguousIncorrectIncorrect

Why Ambiguous exists came out of an early experiment in the paper. With only Correct and Incorrect, CRAG took a completely wrong path on every mistake of the evaluator. Ambiguous is the middle path, where the documents stay and web results come in too. from the paper

The numbers for strips and web search

Refinement uses the same evaluator. A long document is split into strips of a few sentences, the evaluator scores each strip, and of the strips above −0.5 the top 5 stay. from the paper For the milestone question, suppose the syllabus strips got 0.8, 0.1 and −0.7. my scores The first two are above −0.5 and stay, and the Week 10 strip at −0.7 is dropped.

For the web, GPT-3.5 Turbo gets a prompt asking for at most three keywords from the question, separated by commas. The paper's example turns What is Henry Feilden's occupation? into the query Henry Feilden, occupation. The Google Search API returns the top 5 pages, and CRAG puts Wikipedia pages first. The same evaluator then runs on the paragraphs of these pages. from the paper

For our question, such a query would look something like office hour, least crowded, this week. my example But this course's queue log is not on the web, so a web search brings back nothing useful here. CRAG still searches, because it acts only on the evaluator's verdict.

The results: points and percent

The paper added CRAG on top of plain RAG, with SelfRAG-LLaMA2-7b as the generator, the same model that Self-RAG trained: from the paper

TaskRAGCRAGDifference
PopQAaccuracy52.859.8+7.0
BiographyFactScore59.274.1+14.9
PubHealthaccuracy39.075.6+36.6
ARC-Challengeaccuracy53.268.6+15.4

The paper writes these differences as 7.0% accuracy and so on. They are points, and the percentage gain is a different number:

  1. Points: 59.8 − 52.8 = 7.0.
  2. Percentage gain: 7.0 ÷ 52.8 = 0.133, that is 13.3%.

On PubHealth, the 36.6-point gain is 93.8% relative to 39.0, so the accuracy almost doubles. Check: 75.6 ÷ 39.0 = 1.94.

The paper also added CRAG on top of Self-RAG, and calls this Self-CRAG. On PopQA it went from Self-RAG's 54.9 to 61.8, 6.9 points ahead. On ARC it went from 67.3 to 67.2, 0.1 point lower. from the paper

When the generator was plain LLaMA2-hf-7b, with no training for reflection tokens, the Self-RAG method fell to 29.0 on PopQA. With the same generator, plain RAG scored 50.5 and CRAG 54.9. This is the point from step 5: Self-RAG needs its own trained generator, and CRAG works with any generator.

How well the evaluator judges was measured on PopQA. The T5 evaluator was right 84.3% of the time. ChatGPT doing the same job reached 58.0%, 62.4% with a step-by-step prompt, and 64.7% with examples. from the paper

In the ablation, removing refinement brought PopQA from 59.8 to 54.2, removing query rewriting to 56.2, and removing the selection among web results to 58.6. Adding web results every time to plain RAG raised accuracy only from 52.8 to 53.8. So the web alone gives 1.0 point, and CRAG's checking adds 6.0 points on top. from the paper

This checking is not free. In the generation phase, RAG took about 0.363 seconds per question and CRAG 0.512, about 1.41 times as long. The paper says this is a rough estimate that leaves out the time for retrieval and web search. from the paper

9Now read Stanford's slides yourself

Below are Stanford's three agentic RAG slides, unchanged, together with enlarged parts. Under each slide are a few questions. Think of the answer first, then open it. If something stops you, go back to the step named in the answer. If a slide looks small on a phone, click the image to open it at full size.

Stanford slide: agentic RAG, retrieval inside the loop
Stanford's slide: agentic RAG, retrieval inside the loop.

Read the agentic RAG slide

  1. The first line says Everything so far answers one question well. Which parts of this lecture does that cover, and what do they lack?

    Show the answer

    Chunking, BM25, DPR, hybrid search and reranking. All of them improve a single retrieval for a single question. None of them decides whether to retrieve, what to ask, or when to stop. (Step 2)

  2. In Retrieval should be an action the agent takes, what is the action, what is the tool, and what would the observation be?

    Show the answer

    Searching is the action, the retriever or search engine is the tool, and the passages that come back are the observation. The agent reads them and chooses the next action. (Step 1)

  3. The top line mentions when to stop, but the word stop is missing from the three decisions below. Which decision holds it?

    Show the answer

    The third one, whether the result is good enough. If the result is good enough, the agent stops and answers. (Step 2)

Stanford slide: the three steps of Self-RAG
Steps 1 and 2 of the same slide, enlarged
Step 3 of the same slide, enlarged
Stanford's Self-RAG slide, and below it steps 1 and 2, then step 3, enlarged.

Read the Self-RAG slide

  1. Step 1 is called Retrieve on demand. Which decision does on demand refer to, and when does the model retrieve in the paper?

    Show the answer

    Decision 1. The model writes the Retrieve token, and in the paper the system retrieves when yes's share, between yes and no, is above δ = 0.2. (Steps 3 and 6)

  2. In step 3, segment 1 has two green boxes, segment 3 has a green and an orange one, and segment 2 has a red one. Which tokens are these, and how does the score give 1 > 3 > 2?

    Show the answer

    Segment 1 has Relevant and Supported, segment 3 has Relevant and Partially, and segment 2 has Irrelevant. In IsSup, partially gets half weight, so segment 3 scores below segment 1. Segment 2 has a small IsRel share, so it comes last. In our example the same order came from 2.15, 1.475 and 0.425. (Step 6)

  3. The final answer cites passages 1 and 4, although only three passages arrived in step 1. Where did 4 come from? And what do retrieve, critique and generate on the right correspond to?

    Show the answer

    The model wrote Retrieve again, the Repeat part, and the Utah claim came from passage 4 of the next round. The Retrieve token is the retrieval decision, IsRel, IsSup and IsUse are the critique tokens, and generate is the writing of each segment. (Steps 3 and 6)

Stanford slide: the full CRAG diagram
The evaluator and its three verdicts from the same slide, enlarged
The Knowledge Refinement and Knowledge Searching boxes from the same slide, enlarged
The generation row of the same slide, enlarged
Stanford's CRAG slide, and below it the evaluator, both boxes and the generation row, enlarged.

Read the CRAG slide

  1. The evaluator's thought bubble says If retrieved documents are correct to x? What is x here, and in what form does the evaluator answer?

    Show the answer

    x is the question. The evaluator gives each question-document pair a score from −1 to 1, and two thresholds turn the scores into Correct, Ambiguous or Incorrect. The blue box on the left says the same: judge relevance, and pick an action from the confidence. (Steps 4 and 8)

  2. In the Knowledge Refinement box, what are d1, d2, the strips and kin? Why is strip2 a different colour?

    Show the answer

    d1 and d2 are the documents the retriever brought. Decompose splits them into strips, Filter carries the evaluator's icon, and Recompose joins the remaining strips into kin, the internal knowledge. strip2 no longer appears after Filter, so it is the strip that was dropped, like the Week 10 strip of our syllabus. (Steps 4 and 8)

  3. How does the query q in the Knowledge Searching box differ from the question x, and what is kex? Then read the generation row at the bottom.

    Show the answer

    q holds only keywords: Death of a Batman, screenwriter and Wikipedia. The paper's prompt asks for at most three keywords, and CRAG puts Wikipedia pages first. The web search returns k1 to kn, and after Select they become kex, the external knowledge.

    In the generation row, Correct gives the generator x + kin, Ambiguous gives x + kin + kex, and Incorrect gives x + kex. (Steps 4 and 8)

3 points from this lesson

  1. An agent takes an action in a loop, reads the observation, and decides the next action. Every step has three decisions: retrieve or not, which query to send, and whether the result is good enough.
  2. Self-RAG turns the probabilities of each critique kind into shares within that kind, and the weighted sum of the shares is each segment's score. The retrieval decision comes from yes's share and δ = 0.2.
  3. CRAG's T5 evaluator scores from −1 to 1. One document above the upper threshold gives Correct, and all documents below the lower one give Incorrect. On PopQA it lifts RAG from 52.8 to 59.8, which is 7.0 points.

New terms in this lesson

Agent
An LLM that works in a loop: it chooses an action, reads the observation, and decides the next action. For example, when the search for OH timings came back empty, the agent wrote a new query.
Action and tool
What the agent chooses to do, such as searching for OH timings. A tool is the program that does it, such as a search engine.
Observation
The result of an action, which the agent reads before its next step. For example, the result of the first search: every passage scored 0.
Agentic RAG
RAG in which retrieval is an action of the agent, not a fixed step of a pipeline. For example, 0 searches for 17 × 3 and 3 for the Jed Hoyer question.
Self-RAG
The generator itself writes reflection tokens to decide when to retrieve and which segment to keep. For example, Relevant and Partially on the forum post segment.
Reflection tokens
Special tokens added to the vocabulary, which the model writes like any other text. For example, Retrieve, Relevant or Irrelevant, and Supported or Partially.
CRAG
Corrective RAG: an evaluator calls the retrieval Correct, Incorrect or Ambiguous, and refinement, web search or both follow. For example, the forum post at 0.4 is Ambiguous.
Retrieval evaluator
A small model that reads the question and one document and returns a confidence. For example, 0.4 for the forum post at K = 1.
Knowledge refinement
Splitting documents into strips, dropping the useless strips, and joining the rest in order. For example, 2 of the syllabus's 3 strips stay for the milestone question.
Threshold
A limit set in advance, where being above or below it triggers a different action. For example, in CRAG above 0.6 means Correct, and below 0.2 means Incorrect.
Critic
A model that grades what something else wrote or retrieved. For example, in Self-RAG a critic built from Llama 2 7B adds reflection tokens to the training data, and the CRAG slide calls its evaluator a critic.
Segment score (Self-RAG)
The weighted sum of a segment's critique shares: S = 1.0 × sIsRel + 1.0 × sIsSup + 0.5 × sIsUse. For example, the queue log segment's S = 0.9 + 0.875 + 0.5 × 0.75 = 2.15.
Beam search
Moving forward while keeping the B unfinished answers with the highest scores at each step. For example, with B = 2 the queue log (2.15) and forum post (1.475) answers go on, and Homework (0.425) is dropped.