Lesson 2 · Easy version

Retrieval: finding before answering

Retrieval is the step that finds the useful pieces of your documents when a question arrives, so that an LLM can read them and answer. Two parts share the work: the retriever finds the pieces, and the reader reads them and writes the answer.

The need for it showed up in lesson 1. Putting one line of the queue log into the raised the probability of Thursday from 0.21 to 0.81, but we picked that line ourselves. Now picture a company with 2,000 documents of about 30 paragraphs each, so 2,000 × 30 = 60,000 paragraphs. The from lesson 1 had room for 7,000 tokens of passages. At 500 tokens a paragraph, only 7,000 ÷ 500 = 14 paragraphs fit.

So for every question, some program has to pick the 14 out of 60,000 that hold the answer. This lesson builds such a program in the simplest way: by counting words. Of the course's 3 questions, it handles the first well. On the second it ties a right passage with a wrong one, and on the third it returns nothing.

The ladder for this lesson

The ladder has 4 steps. Under each step you can see what you will be able to do after it.

  1. Corpus, document, passage and queryExplain why a retriever does not return whole documents.
  2. Retrieval: the right passage into the prompt when a question arrivesRead a retrieval prompt and Stanford's Why Retrieval? slide.
  3. The retriever: score every passage, then take the top KCompute scores by hand, and explain why the old forum post ties with the queue log.
  4. The recall ceiling: a passage that never arrives is never readShow with arithmetic why the retriever sets the limit for the whole system.

What you need first: Lesson 1 (token, prompt, context window, parameters, hallucination). This lesson explains everything else it uses. If you already know a step, skip to the next one. A word with a dotted underline and ▾ opens its definition when you click it.

1Corpus, document and passage: where the retriever looks

The set of all your documents is called the corpus, and the retriever searches inside it. Each single item in the corpus is a document: a web page, a file or an email. Another name for the corpus is the collection.

The retriever does not return a whole document, because the prompt has room for only 14 pieces, and most of a long document has nothing to do with the question. The milestone date in a 40-page syllabus probably takes 2 lines. So the retriever returns a small piece, such as one paragraph, and this piece is called a passage.

The question you are searching for is the query. This lesson's first query is the one from lesson 1: which office hour is least crowded.

Later lessons keep coming back to this small corpus. It has 6 documents, with one passage from each. Every passage also has a short ID, such as P3, so that an answer can name it. the syllabus, queue log, Homework 1 announcement and project page come from the slides, the office hours page and the old forum post are mine

IDPassage
P1Grading. The project milestone is due Week 6, Thursday 11:59pm. Late days may not be applied to the milestone. The final report is due Week 10.Syllabus
P2Office hours: Tuesday 4pm in Room 104 and Thursday 9am on Zoom, run by the TAs.Office hours page
P3Queue log, last 3 weeks: Tuesday 4pm office hour averaged 14 students waiting; Thursday 9am office hour averaged 3.Queue log
P4Old forum post (2024): midweek afternoons are usually the least crowded time to visit.Old forum post
P5Homework 1 will be released on Oct 5.Homework 1 announcement
P6Project proposal due on Oct 9. Teams of 1-3 people are recommended.Project page

Think first

Suppose the syllabus and the Homework 1 announcement were two paragraphs of the same document. How many documents would there be, and how many passages?

Show the answer

5 documents, but still 6 passages, and the retriever chooses among those 6. The company has the same split: 2,000 documents, but 60,000 passages.

2Retrieval: putting the right passage into the prompt when a question arrives

Retrieval means finding the most useful passages in the corpus when a query arrives, and putting them into the prompt with the question. The model then does not have to memorize everything in advance. Stanford's slide calls this just-in-time: the right text reaches the model exactly when the question does.

For our query the retriever finds the queue log, and the LLM gets a prompt like this: my example

Answer the question using the passage below. Cite the passage ID.
[P3] Queue log, last 3 weeks: Tuesday 4pm office hour averaged 14 students waiting; Thursday 9am office hour averaged 3.
Question: Which office hour is least crowded?

The answer comes back as Thursday 9am, about 3 students ahead of you [P3]. The [P3] at the end says which passage the claim came from. A source written next to an answer is called a citation. With the citation you can open the queue log and check that it really says this.

Stanford slide: Why Retrieval? The answer of an LLM alone and of an LLM with retrieval
Stanford's Why Retrieval? slide. On the left is the LLM's guess on its own. On the right is the queue log the retriever returned, and the answer written from it. Click the image to open it at full size.

In lesson 1 all of the model's knowledge sat in its , which caused two problems. Adding a new fact needed training, and the model could not name its source. With retrieval you add a new fact by adding a passage to the corpus, and the citation shows the source. Stanford's slide calls these two benefits Dynamic and Interpretable.

Stanford slide: using retrieval to overcome the shortcomings of LMs, Dynamic and Interpretable
Stanford's slide: just-in-time, Dynamic and Interpretable. In the diagram at the bottom, the [1] in the reader's answer is a citation.

Lesson 1 called knowledge stored inside the parameters . Here the knowledge sits in the corpus, outside the model, and knowledge kept this way is called non-parametric memory.

Retrieval is only as good as the search, though. The end of lesson 1 showed that with the old forum post in the prompt, the model may write Wednesday again. So the next step builds a firm rule for searching.

3The retriever: score every passage, then take the top K

A retriever gives every passage a score for the query, sorts the passages by score, and returns the top K. A score is a number that says how useful a passage is for the query.

The K highest-scoring passages are called the top-K. K is fixed in advance, for example 2 or 100.

The simplest score is word overlap: how many of the query's words also appear in the passage. But words such as which, is and the appear in almost every text, so counting them tells you nothing.

Very common words like these are called stopwords, and the retriever removes them before counting. Remove which and is from the query which office hour is least crowded. Four words are left: office, hour, least and crowded.

Think first

The queue log holds the answer, and the old forum post holds only an opinion from 2024. Look for the four words in both passages in the corpus table. Which one gets the higher word-overlap score?

Show the answer

Neither. Both get 2. The queue log has office and hour, and the forum post has least and crowded.

Here are the scores for the whole corpus:

PassageWords matchedScore
Queue logoffice, hour2
Old forum postleast, crowded2
Office hours pageofficeThe retriever matches words exactly, so hours and hour are different words to it.1
Syllabus, Homework 1 announcement, project pagenone0

Word overlap only counts words, so it cannot see that the queue log is three weeks of data and the forum post a 2024 opinion. With K = 2 the retriever returns both. The prompt then carries the old opinion next to the real data, and the model may write Wednesday again because of it.

The part that reads the query and the top-K passages and writes the answer is called the reader. Today the reader is usually an LLM that was never trained for this particular task and simply follows the instruction in the prompt. The first line of the prompt above, Answer the question using the passage below, is that instruction.

Doing a task from an instruction alone, without training for it, is called zero-shot. Since the reader is usually such a ready-made LLM, lessons 3 to 15 mostly work on making the retriever better.

Run the retriever yourself

Pick a query and change K. The retriever does not return passages with score 0. When scores tie, the passage listed first in the corpus comes first.

2

RankPassageScore

Work it out

Query: when is the project milestone due. Remove the stopwords and score the syllabus and the project page. With K = 1, which passage does the reader get? Check it in the widget.

Show the answer

The words left are project, milestone and due. The syllabus has all three, so its score is 3. The project page has project and due, so its score is 2. With K = 1 the reader gets the syllabus, which has the right date: Week 6, Thursday 11:59pm.

4The recall ceiling: a passage that never arrives is never read

Now pick the third query in the widget: when can I get help from a TA. After removing stopwords, help and ta are left. The answer is in the office hours page, but that page says TAs, not ta, and no passage contains help. So every score is 0 and the retriever returns nothing. However good the reader is, the office hours page never reaches its prompt.

How often a retriever finds the passage that is needed is, roughly, the retriever's recall. If the right passage reaches the top K for 60 out of 100 questions, the recall is 60%. 60% means 60 out of 100, that is 60 ÷ 100 = 0.6. Lesson 13 shows how to measure recall precisely.

Now add the reader: when it gets the right passage, it answers 90% of questions correctly, that is 0.9 per question. On the 60 questions that gives 60 × 0.9 = 54 correct answers. On the other 100 − 60 = 40 questions, no correct answer can come from the passages. You can now improve either the reader or the retriever. my example

ChangeArithmeticCorrect answers
As it is now60 × 0.954
Make the reader perfect60 × 1.060
Raise the retriever to 80, same reader??

Work out the third row

The retriever now brings the right passage for 80 questions, and the reader is still right 90% of the time. How many correct answers do you get? More or fewer than with the perfect reader?

Show the answer

80 × 0.9 = 72, which is 72 − 60 = 12 more than the perfect reader. Even a perfect reader cannot go past 60, because for the other 40 questions the right passage never reached it.

The perfect reader stops at 60 because of a rule of multiplication. Multiply a number by something smaller than 1 and it shrinks, as in 60 × 0.5 = 30. Multiply it by 1 and it stays the same, as in 60 × 1.0 = 60. The reader's share can never be above 1, because 1 means every question answered correctly.

So the correct answers can never exceed the 60 questions for which the retriever brought the right passage. The retriever's recall becomes the upper limit of the whole system.

This limit is called the recall ceiling. Stanford's slide writes Missing evidence limits agent under it: evidence that never reaches the prompt holds the system back. Agents come in lesson 16.

The retrieval part of Stanford's Issues with RAG slide: Recall ceiling, Missing evidence limits agent
The retrieval part of Stanford's slide Issues with RAG. The other two problems come in lesson 18.

A larger K raises recall, but wrong passages come along too, like the old forum post at K = 2. And the company's prompt cannot hold more than 14 passages anyway.

Word overlap failed twice, even on 6 passages. The first time it ranked a right and a wrong passage equally. The second time the query and the passage said the same thing in different words, help from a TA and run by the TAs, and the score was 0. Lesson 3 starts right here: turning text into numbers so that texts saying similar things get numbers close to each other.

What the dense version adds

The dense version contains this whole lesson, followed by 4 more steps that take you to the level of the Stanford slides:

  • Step 5: the symbols on the retriever-reader slide, such as f(𝒟, Q), and the maths of functions.
  • Step 6: DrQA, a real retriever-reader system that searches all of Wikipedia, and its training data.
  • Step 7: the recall ceiling in DrQA's real numbers, and the full arithmetic of changing K on our corpus.
  • Step 8: this lesson's Stanford slides, which you read yourself.

Open the dense version

3 points from this lesson

  1. Retrieval: for every query, pick a few passages from the corpus and put them into the prompt. It needs no training, and the answer comes with a citation.
  2. The retriever scores every passage and returns the top K, and the reader reads them and writes the answer. Word overlap gives the queue log and the old forum post the same score, 2.
  3. A passage the retriever did not bring is never read by the reader. So the retriever's recall is the upper limit of the whole system: the recall ceiling.

New terms in this lesson

Corpus (collection)
The set of all documents. For example, the company's 2,000 documents, or this course's 6 short texts.
Document
One item in the corpus: a web page, a file or an email. For example, the course syllabus.
Passage
A small piece of a document, which the retriever returns. For example, the syllabus paragraph that holds the milestone date.
Query
The question you search for. For example, “which office hour is least crowded”.
Retrieval
Finding useful passages for a query and putting them into the prompt. For example, the queue log going into the prompt with the question.
Citation
The source written next to an answer. For example, the [P3] in “Thursday 9am [P3]”, which points to the queue log.
Non-parametric memory
Knowledge kept outside the model, in the corpus. For example, the queue log: changing it needs no training.
Retriever
The part that scores every passage and returns the top K. For example, this lesson's word-overlap retriever.
Score
A number for how useful a passage is for the query. For example, the queue log's word-overlap score of 2.
Top-K
The K passages with the highest scores. For example, the queue log and the old forum post at K = 2.
Word overlap
How many of the query's words appear in the passage. The simplest score. For example, the queue log has office and hour, so its score is 2.
Stopwords
Very common words such as is, the and which, left out of the count. From “which office hour is least crowded”, which and is are removed.
Reader
The part that reads the query and the top-K passages and writes the answer. Today usually an LLM. For example, reading the queue log and writing “Thursday 9am”.
Zero-shot
Doing a task from an instruction alone, without training on that task. For example, the prompt line “Answer the question using the passage below”.
Recall
How often the retriever finds the passage that is needed. For example, 60 out of 100 questions gives a recall of 60%.
Recall ceiling
The retriever's recall becomes the upper limit of the whole system. With a recall of 60%, no reader can get more than 60 of 100 answers from the passages.