Lesson 2 · Dense version

Retrieval: finding before answering

Retrieval is the step that finds the useful pieces of your documents when a question arrives, so that an LLM can read them and answer. Two parts share the work: the retriever finds the pieces, and the reader reads them and writes the answer.

The need for it showed up in lesson 1. Putting one line of the queue log into the raised the probability of Thursday from 0.21 to 0.81, but we picked that line ourselves. Now picture a company with 2,000 documents of about 30 paragraphs each, so 2,000 × 30 = 60,000 paragraphs. The from lesson 1 had room for 7,000 tokens of passages. At 500 tokens a paragraph, only 7,000 ÷ 500 = 14 paragraphs fit.

So for every question, some program has to pick the 14 out of 60,000 that hold the answer. This lesson builds such a program in the simplest way: by counting words. Of the course's 3 questions, it handles the first well. On the second it ties a right passage with a wrong one, and on the third it returns nothing.

The ladder for this lesson

The ladder has 8 steps. The first 4 are the same as in the easy version, and steps 5 to 8 take you to the level of the Stanford slides. Under each step you can see what you will be able to do after it.

  1. Corpus, document, passage and queryExplain why a retriever does not return whole documents.
  2. Retrieval: the right passage into the prompt when a question arrivesRead a retrieval prompt and Stanford's Why Retrieval? slide.
  3. The retriever: score every passage, then take the top KCompute scores by hand, and explain why the old forum post ties with the queue log.
  4. The recall ceiling: a passage that never arrives is never readShow with arithmetic why the retriever sets the limit for the whole system.
  5. From here: the Stanford level
  6. Retriever and reader as functionsTie f(𝒟, Q) and g(Q, {P1, …, PK}) to your own example, and explain every line of the slide.
  7. DrQA: a retriever and a reader over all of WikipediaExplain DrQA's retriever, its reader and its distantly supervised training data.
  8. The recall ceiling in real numbers, and the arithmetic of KWork out the ceiling from DrQA's numbers, and give the result of every K on our corpus.
  9. Read Stanford's slides yourselfExplain every line of this lesson's Stanford slides.

What you need first: Lesson 1 (token, prompt, context window, parameters, hallucination). This lesson explains everything else it uses. From step 5 on, a few terms from later lessons appear, such as TF-IDF and Exact Match, and this lesson explains them briefly where they come up. If you already know a step, skip to the next one. A word with a dotted underline and ▾ opens its definition when you click it.

1Corpus, document and passage: where the retriever looks

The set of all your documents is called the corpus, and the retriever searches inside it. Each single item in the corpus is a document: a web page, a file or an email. Another name for the corpus is the collection.

The retriever does not return a whole document, because the prompt has room for only 14 pieces, and most of a long document has nothing to do with the question. The milestone date in a 40-page syllabus probably takes 2 lines. So the retriever returns a small piece, such as one paragraph, and this piece is called a passage.

The question you are searching for is the query. This lesson's first query is the one from lesson 1: which office hour is least crowded.

Later lessons keep coming back to this small corpus. It has 6 documents, with one passage from each. Every passage also has a short ID, such as P3, so that an answer can name it. the syllabus, queue log, Homework 1 announcement and project page come from the slides, the office hours page and the old forum post are mine

IDPassage
P1Grading. The project milestone is due Week 6, Thursday 11:59pm. Late days may not be applied to the milestone. The final report is due Week 10.Syllabus
P2Office hours: Tuesday 4pm in Room 104 and Thursday 9am on Zoom, run by the TAs.Office hours page
P3Queue log, last 3 weeks: Tuesday 4pm office hour averaged 14 students waiting; Thursday 9am office hour averaged 3.Queue log
P4Old forum post (2024): midweek afternoons are usually the least crowded time to visit.Old forum post
P5Homework 1 will be released on Oct 5.Homework 1 announcement
P6Project proposal due on Oct 9. Teams of 1-3 people are recommended.Project page

Think first

Suppose the syllabus and the Homework 1 announcement were two paragraphs of the same document. How many documents would there be, and how many passages?

Show the answer

5 documents, but still 6 passages, and the retriever chooses among those 6. The company has the same split: 2,000 documents, but 60,000 passages.

2Retrieval: putting the right passage into the prompt when a question arrives

Retrieval means finding the most useful passages in the corpus when a query arrives, and putting them into the prompt with the question. The model then does not have to memorize everything in advance. Stanford's slide calls this just-in-time: the right text reaches the model exactly when the question does.

For our query the retriever finds the queue log, and the LLM gets a prompt like this: my example

Answer the question using the passage below. Cite the passage ID.
[P3] Queue log, last 3 weeks: Tuesday 4pm office hour averaged 14 students waiting; Thursday 9am office hour averaged 3.
Question: Which office hour is least crowded?

The answer comes back as Thursday 9am, about 3 students ahead of you [P3]. The [P3] at the end says which passage the claim came from. A source written next to an answer is called a citation. With the citation you can open the queue log and check that it really says this.

Stanford slide: Why Retrieval? The answer of an LLM alone and of an LLM with retrieval
Stanford's Why Retrieval? slide. On the left is the LLM's guess on its own. On the right is the queue log the retriever returned, and the answer written from it. Click the image to open it at full size.

In lesson 1 all of the model's knowledge sat in its , which caused two problems. Adding a new fact needed training, and the model could not name its source. With retrieval you add a new fact by adding a passage to the corpus, and the citation shows the source. Stanford's slide calls these two benefits Dynamic and Interpretable.

Stanford slide: using retrieval to overcome the shortcomings of LMs, Dynamic and Interpretable
Stanford's slide: just-in-time, Dynamic and Interpretable. In the diagram at the bottom, the [1] in the reader's answer is a citation.

Lesson 1 called knowledge stored inside the parameters . Here the knowledge sits in the corpus, outside the model, and knowledge kept this way is called non-parametric memory.

Retrieval is only as good as the search, though. The end of lesson 1 showed that with the old forum post in the prompt, the model may write Wednesday again. So the next step builds a firm rule for searching.

3The retriever: score every passage, then take the top K

A retriever gives every passage a score for the query, sorts the passages by score, and returns the top K. A score is a number that says how useful a passage is for the query.

The K highest-scoring passages are called the top-K. K is fixed in advance, for example 2 or 100.

The simplest score is word overlap: how many of the query's words also appear in the passage. But words such as which, is and the appear in almost every text, so counting them tells you nothing.

Very common words like these are called stopwords, and the retriever removes them before counting. Remove which and is from the query which office hour is least crowded. Four words are left: office, hour, least and crowded.

Think first

The queue log holds the answer, and the old forum post holds only an opinion from 2024. Look for the four words in both passages in the corpus table. Which one gets the higher word-overlap score?

Show the answer

Neither. Both get 2. The queue log has office and hour, and the forum post has least and crowded.

Here are the scores for the whole corpus:

PassageWords matchedScore
Queue logoffice, hour2
Old forum postleast, crowded2
Office hours pageofficeThe retriever matches words exactly, so hours and hour are different words to it.1
Syllabus, Homework 1 announcement, project pagenone0

Word overlap only counts words, so it cannot see that the queue log is three weeks of data and the forum post a 2024 opinion. With K = 2 the retriever returns both. The prompt then carries the old opinion next to the real data, and the model may write Wednesday again because of it.

The part that reads the query and the top-K passages and writes the answer is called the reader. Today the reader is usually an LLM that was never trained for this particular task and simply follows the instruction in the prompt. The first line of the prompt above, Answer the question using the passage below, is that instruction.

Doing a task from an instruction alone, without training for it, is called zero-shot. Since the reader is usually such a ready-made LLM, lessons 3 to 15 mostly work on making the retriever better.

Run the retriever yourself

Pick a query and change K. The retriever does not return passages with score 0. When scores tie, the passage listed first in the corpus comes first.

2

RankPassageScore

Work it out

Query: when is the project milestone due. Remove the stopwords and score the syllabus and the project page. With K = 1, which passage does the reader get? Check it in the widget.

Show the answer

The words left are project, milestone and due. The syllabus has all three, so its score is 3. The project page has project and due, so its score is 2. With K = 1 the reader gets the syllabus, which has the right date: Week 6, Thursday 11:59pm.

4The recall ceiling: a passage that never arrives is never read

Now pick the third query in the widget: when can I get help from a TA. After removing stopwords, help and ta are left. The answer is in the office hours page, but that page says TAs, not ta, and no passage contains help. So every score is 0 and the retriever returns nothing. However good the reader is, the office hours page never reaches its prompt.

How often a retriever finds the passage that is needed is, roughly, the retriever's recall. If the right passage reaches the top K for 60 out of 100 questions, the recall is 60%. 60% means 60 out of 100, that is 60 ÷ 100 = 0.6. Lesson 13 shows how to measure recall precisely.

Now add the reader: when it gets the right passage, it answers 90% of questions correctly, that is 0.9 per question. On the 60 questions that gives 60 × 0.9 = 54 correct answers. On the other 100 − 60 = 40 questions, no correct answer can come from the passages. You can now improve either the reader or the retriever. my example

ChangeArithmeticCorrect answers
As it is now60 × 0.954
Make the reader perfect60 × 1.060
Raise the retriever to 80, same reader??

Work out the third row

The retriever now brings the right passage for 80 questions, and the reader is still right 90% of the time. How many correct answers do you get? More or fewer than with the perfect reader?

Show the answer

80 × 0.9 = 72, which is 72 − 60 = 12 more than the perfect reader. Even a perfect reader cannot go past 60, because for the other 40 questions the right passage never reached it.

The perfect reader stops at 60 because of a rule of multiplication. Multiply a number by something smaller than 1 and it shrinks, as in 60 × 0.5 = 30. Multiply it by 1 and it stays the same, as in 60 × 1.0 = 60. The reader's share can never be above 1, because 1 means every question answered correctly.

So the correct answers can never exceed the 60 questions for which the retriever brought the right passage. The retriever's recall becomes the upper limit of the whole system.

This limit is called the recall ceiling. Stanford's slide writes Missing evidence limits agent under it: evidence that never reaches the prompt holds the system back. Agents come in lesson 16.

The retrieval part of Stanford's Issues with RAG slide: Recall ceiling, Missing evidence limits agent
The retrieval part of Stanford's slide Issues with RAG. The other two problems come in lesson 18.

A larger K raises recall, but wrong passages come along too, like the old forum post at K = 2. And the company's prompt cannot hold more than 14 passages anyway.

Word overlap failed twice, even on 6 passages. The first time it ranked a right and a wrong passage equally. The second time the query and the passage said the same thing in different words, help from a TA and run by the TAs, and the score was 0. Lesson 3 starts right here: turning text into numbers so that texts saying similar things get numbers close to each other.

From here: the Stanford level

The 4 steps above are the same as in the easy version. The 4 steps below take you to the level of the Stanford slides: the symbols on the retriever-reader slide and the maths of functions, then a real system called DrQA and its training data, then the recall ceiling in real numbers, and finally reading the slides yourself.

5Retriever and reader as functions: the slide's symbols

Stanford's slide writes the retriever and the reader as two functions: the retriever turns the corpus and the query into K passages, and the reader turns the query and those passages into an answer. To read it you first need what a function is.

A function is a rule that takes an input and gives an output. If the rule is “double it and add 1”, you write it as f(x) = 2x + 1. Here f is the name of the rule and x is the input, so f(3) = 2 × 3 + 1 = 7. A function in code does the same thing: it takes arguments and returns a value.

A function can also take more than one input. In f(x, y) = x + 2y, f(1, 3) = 1 + 2 × 3 = 7. Both lines on the slide are functions with two inputs, and “⟶” means “gives as output”:

Retrieverf(𝒟, Q) ⟶ P1, …, PK
Readerg(Q, {P1, …, PK}) ⟶ A

The table ties each symbol to our example:

SymbolMeaningIn our example
𝒟 = D1, D2, …, DNThe whole corpus, a list of documents. The small number below says which document in the list it is, and “…” means “and so on up to”.D1 = syllabus up to D6 = project page
NThe number of documents in the corpus62,000 at the company
QThe querywhich office hour is least crowded
fThe retriever's ruleword overlap, then top-K
KHow many passages to return. The slide says K is fixed in advance, for example 100.2
P1, …, PKThe K passages the retriever returns, in order of scoreP1 = queue log, P2 = old forum post
{ … }Curly brackets make a group of things, called a .{queue log, old forum post}
gThe reader's rulethe zero-shot LLM from step 3
AThe answer, which the slide calls an answer string, meaning a piece of textThursday 9am, about 3 students ahead of you

The input has documents (D) and the output has passages (P), because, as step 1 said, a retriever returns pieces of documents. On our query the two functions run in three steps:

  1. f(𝒟, Q): the retriever scores every passage. In corpus order the scores are 0, 1, 2, 2, 0, 0.
  2. With K = 2 the top two passages come back: P1 = queue log and P2 = old forum post. Both score 2, so corpus order decided here.
  3. g(Q, {queue log, old forum post}): the reader reads both passages and writes A. The set holds two passages that contradict each other, so A may say Thursday 9am or Wednesday.

Think first

For the query when is the project milestone due and K = 2, what does f(𝒟, Q) return? With K = 1, what is the input to g?

Show the answer

With K = 2, f returns P1 = syllabus (score 3) and P2 = project page (score 2). With K = 1 the reader gets g(Q, {syllabus}), and A = Week 6, Thursday 11:59pm.

A reading comprehension problem: the reader's job

A reading comprehension problem! is what the slide writes next to the reader line. Reading comprehension means answering a question about a text you are given. After the retriever has run, that is exactly the reader's job, because the passages are already in its hands.

The best-known dataset for this task is SQuAD, the Stanford Question Answering Dataset. Its makers showed people a Wikipedia paragraph, and the people wrote questions about that paragraph. The answer is always a piece of the same paragraph, and such a piece is called a span. The training part has 87,599 questions. from the paper One example:

Paragraph: … At the beginning of the 17th century the empire contained 32 provinces and numerous vassal states. …
Q: How many provinces did the Ottoman empire contain in the 17th century?
A: 32

The rest of the slide: a fixed retriever, and two kinds of reader

The slide has two more lines at the bottom. The first is about the retriever: A standard TF-IDF information-retrieval sparse model (a fixed module). It has three parts:

The second line is about the reader: a neural reading comprehension model. A neural model is a model with parameters that learns from examples, like the LLM in lesson 1. The slide gives two routes for such a reader:

6DrQA: a retriever and a reader over all of Wikipedia

DrQA is a 2017 system that answers a question by searching all of English Wikipedia. Danqi Chen and colleagues built it at Facebook AI Research. from the paper Stanford's slide shows it as an example of the retriever-reader design. The slide's question is How many of Warsaw's inhabitants spoke Polish in 1933?, and the answer is 833,500.

Here the question can be about any topic, and the system is not handed the right paragraph. Question answering like this is called open-domain QA. In SQuAD-style reading comprehension the paragraph comes with the question. In open-domain QA the system has to find it in the whole corpus itself.

DrQA's retriever

DrQA's corpus is English Wikipedia from 21 December 2016. After some pages such as lists were removed, 5,075,182 articles were left, so N = 5,075,182. The retriever is not trained, and the paper itself calls it non-machine learning. It treats the question and every article as lists of TF-IDF word weights and scores every article from these lists. It returns 5 articles per question, so K = 5. from the paper

Besides single words, this retriever also matches pairs of words that appear side by side. Two adjacent words are called a bigram. The bigrams of our query are which office, office hour, hour is, is least and least crowded.

These articles hold 9,008,962 different words, and far more different bigrams. So DrQA uses a hash function to give each bigram the number of one of 224 buckets. A hash function is a fixed rule that turns any text into a number within a fixed range. 224 means 2 multiplied by itself 24 times, which is 16,777,216 buckets. from the paper

Think first

On our corpus, count the query's bigrams instead of its words. Does the tie between the queue log and the old forum post break?

Show the answer

No. The queue log has office hour and the forum post has least crowded, so each matches 1 bigram. The other 4 passages share no bigram with the query. Making bigrams after removing stopwords (office hour, hour least, least crowded) gives the same result. my arithmetic

The paper measured for what percentage of questions the answer text appears in at least one of the 5 articles the retriever returns. Wikipedia's own search engine was the comparison: from the paper

DatasetWikipedia searchDrQA, words onlyDrQA, words + bigrams
SQuAD62.776.177.8
CuratedTREC2,180 questions from the TREC competitions81.085.286.0
WebQuestionsquestions taken from Google Suggest73.775.574.4
WikiMoviesquestions about films61.754.470.3

DrQA's reader

DrQA's reader is a neural network trained on SQuAD. It gives every token of a paragraph two probabilities: that the answer starts here, and that the answer ends here. It then picks the piece whose start and end probabilities have the largest product, with no more than 15 tokens between start and end. from the paper

So DrQA's reader does not make up an answer: it picks a span from the text it was given. Step 7 needs this fact for the ceiling. On SQuAD questions that were not used in training, with the right paragraph given, this reader's is 69.5%. Exact Match counts an answer as correct only if it matches the right answer exactly. from the paper

Distant supervision: training data without paragraph labels

Distant supervision means building training examples for data that has only questions and answers, by finding the right paragraph automatically. SQuAD gives every question its paragraph, but CuratedTREC, WebQuestions and WikiMovies give only question-answer pairs. The reader has to learn to pick a span from a paragraph, so it needs paragraphs. This is what the slide means by distantly-supervised QA datasets.

Take one pair on our corpus: the question when is the project milestone due and the answer Week 6, Thursday 11:59pm. No passage carries a label. my example

  1. Run the retriever on the question: the syllabus gets 3, the project page 2, and the other four get 0.
  2. Of the returned passages, drop those that do not contain the answer text exactly. The project page does not contain Week 6, Thursday 11:59pm, so it goes.
  3. The syllabus that is left becomes a training example: the question, the syllabus, and the answer span inside it.

DrQA did this at scale and added a few filters. For each question it took the retriever's top 5 articles and dropped paragraphs without an exact match of the answer. It also dropped paragraphs shorter than 25 or longer than 1,500 characters. If the question contained a name, such as Chadwick, paragraphs without that name were dropped too. It then counted the question's words and bigrams in a 20-token window around the answer and kept at most 5 paragraphs. from the paper

No person labelled the paragraph. The label came automatically from the answer text, which is why this is called distant, meaning indirect, supervision. An answer string can also appear in a paragraph by coincidence, so such a label is sometimes wrong. my caution

7The recall ceiling in real numbers, and the full arithmetic of K

DrQA shows the recall ceiling clearly, because its reader can only pick a span from the text it was given. If the answer text is not in the 5 articles at all, the reader cannot produce it in any way. The retriever brings an article with the answer for 77.8% of SQuAD questions, so the whole system can be right on at most 77.8% of them. numbers from the paper, the ceiling arithmetic is mine

The paper measured the reader in several situations. On the SQuAD questions the results are: from the paper

What the reader gotExact Match (%)
Only the right paragraph, as in SQuAD69.5
The whole right article49.4
The retriever's 5 articles, from all of Wikipedia29.8 with a reader trained on SQuAD and the distantly supervised data together27.1

The ceiling, the percentage of questions for which the retriever brought an article with the answer: 77.8.

Now apply the multiplication from step 4. If the reader were as good with 5 articles as with one right paragraph, the correct answers would be 0.695 of the 77.8%. my arithmetic

  1. Turn both percentages into shares: 77.8% = 0.778 and 69.5% = 0.695.
  2. Multiply: 0.778 × 0.695 = 0.541, which is 54.1%.
  3. Compare with the real result: 54.1 ÷ 27.1 = 2.0, so the real result is half of this estimate.

The paper gives two reasons for the drop. First, the reader has to read 5 whole articles instead of one paragraph, and many sentences on the same topic look like answers. That is why even the right article brings the number down from 69.5 to 49.4. Second, people wrote the SQuAD questions while looking at one specific paragraph, so without that paragraph many questions read as incomplete. from the paper

So the recall ceiling is only an upper limit: the system can land anywhere below it. In DrQA the ceiling is 77.8 and the real result is 27.1. The questions the retriever missed set the ceiling. The reader's weakness makes the gap between the ceiling and the real result.

Changing K on our corpus

Here is the result for every K on the least-crowded query. Passages with score 0 are not returned. my arithmetic

KWhat the retriever returnsQueue log in?Without the answer
1queue logyes0
2queue log, old forum postyes1
3 to 6queue log, old forum post, office hours pageyes2

At K = 1 the queue log comes back only because ties follow corpus order, and the queue log (P3) is listed before the forum post (P4). If the forum post had been listed first, the reader would get only the forum post at K = 1. So in a tie the result is decided by a rule that has nothing to do with the question.

Now look at all three widget queries at K = 2. The least-crowded query brings the queue log, the milestone query brings the syllabus, and the TA query brings nothing. So the right passage arrives for 2 of 3 queries, and 2 ÷ 3 = 0.667. Lesson 13 calls a measure of this kind . On these 3 queries no reader can answer more than 2 of 3 from the passages.

Think first

Set K = 6 for the TA query. Does the office hours page reach the reader now?

Show the answer

No. Every passage scores 0, and the retriever never returns passages with score 0, so nothing comes back whatever K is. A larger K raises recall only when the right passage scores above 0 but sits lower in the ranking.

In the company's system every passage is 500 tokens, so K passages need K × 500 tokens. At K = 14 that is 14 × 500 = 7,000 tokens, the whole space from lesson 1. A larger K does not fit into the prompt, however much it would help recall.

8Now read Stanford's slides yourself

Below are this lesson's Stanford slides, unchanged. Under each slide are a few questions. Think of your own answer first, then open it. If something is unclear, go back to the step named in the answer. If a slide looks small on a phone, click the image and it opens at full size.

Stanford slide: Why Retrieval?
Stanford's slide: Why Retrieval?

Read the Why Retrieval? slide

  1. The slide's query also contains this week: which office hour this week is least crowded. What changes in the word-overlap scores?

    Show the answer

    this is a stopword, and week is left. The syllabus has Week 6 and Week 10, so the syllabus gets 1. The queue log says weeks, not week, so it stays at 2, and the forum post stays at 2 as well. The tie stays. (Step 3)

  2. In the box on the right, which part is the retriever's work and which is the reader's?

    Show the answer

    The Retrieved line is the passage the retriever returned, the queue log. The answer below it is what the reader wrote from that passage. (Steps 2 and 3)

  3. The answer on the left talks about midweek hours. Which passage in our corpus says the same, and what does it do at K = 2?

    Show the answer

    The old forum post: midweek afternoons are usually the least crowded. At K = 2 it goes into the prompt with the queue log, and the reader may write Wednesday again. (Step 3)

Stanford slide: Using Retrieval to Overcome LMs’ Shortcomings
The diagram of the same slide, enlarged
Stanford's slide: Using Retrieval to Overcome LMs’ Shortcomings, with its diagram enlarged below.

Read the just-in-time slide

  1. What does just-in-time mean, and how is it different from memorize everything?

    Show the answer

    The model does not keep everything in its parameters. The right text enters the prompt exactly when the question arrives. (Step 2)

  2. Which problems from lesson 1 do Dynamic and Interpretable solve?

    Show the answer

    Dynamic: a new fact needs no training, only a new passage in the corpus. Interpretable: with the citation you can open the source and check it. (Step 2)

  3. In the diagram, what is the Text Collection, and why does the answer end with [1]?

    Show the answer

    The Text Collection is the corpus the retriever searches. [1] Wikipedia - Immune system is the source of the returned passage, and the [1] in the answer is its citation. (Steps 1 and 2)

Stanford slide: Retrieval Often Used in Open-Domain QA, DrQA
Stanford's slide: Retrieval Often Used in Open-Domain QA. The Wikipedia article in the middle is only a picture, and you do not need to read it.

Read the open-domain QA slide

  1. What is the corpus on this slide, and what is its N in DrQA?

    Show the answer

    All of English Wikipedia. The paper has 5,075,182 articles, so N = 5,075,182. (Step 6)

  2. Where did the Document Reader get 833,500 from?

    Show the answer

    From the Warsaw article the retriever returned. DrQA's reader does not make up an answer: it picks a span from the article. (Step 6)

  3. Why is this called open-domain QA?

    Show the answer

    The question can be about any topic, and the system is not handed the paragraph. It has to find it in all of Wikipedia itself. (Step 6)

Stanford slide: Retriever-Reader Framework
The retriever and reader lines of the same slide, enlarged
The retriever and reader text of the same slide, enlarged
Stanford's slide: Retriever-Reader Framework, with its two lines and the text at the bottom enlarged below.

Read the Retriever-Reader Framework slide

  1. Write f(𝒟, Q) ⟶ P1, …, PK for our first query and K = 2.

    Show the answer

    f(the 6-passage corpus, which office hour is least crowded) ⟶ queue log, old forum post. (Step 5)

  2. Why does the slide call the retriever a fixed module?

    Show the answer

    DrQA's TF-IDF retriever is not trained. Its rule is fixed in advance, like the rule of our word overlap. (Steps 5 and 6)

  3. What does distantly-supervised mean in the reader's first bullet?

    Show the answer

    For datasets with only questions and answers, a rule found the paragraphs automatically, from the retriever and the answer text. On our corpus that rule picks the syllabus for the milestone question. (Step 6)

Stanford slide: Issues with RAG
The retrieval part of the same slide, enlarged
Stanford's slide: Issues with RAG, with its retrieval part enlarged below. The other problems come in lesson 18.

Read the Issues with RAG slide

  1. Missing evidence limits agent: what was this limit in the table of step 4?

    Show the answer

    60 questions. The retriever brought the right passage for 60 questions, so even the perfect reader stopped at 60. (Step 4)

  2. What is this limit in DrQA, and what is the real result?

    Show the answer

    The ceiling is 77.8%, and the real Exact Match is 27.1%. (Step 7)

  3. For the least-crowded query at K = 2, the prompt holds two passages that contradict each other. Which reader-side boxes does this match?

    Show the answer

    Distraction, meaning similar but useless text, and Evidence conflict, meaning sources that disagree. Both come in lesson 18. (Step 3)

Stanford slide: Announcements
Stanford slide: Project Team and Key Considerations
Stanford slide: Chunking: Same Text, Three Splits
The three methods of the chunking slide, enlarged
Three Stanford slides that our corpus passages come from: Announcements, Project Team and Key Considerations, and Chunking: Same Text, Three Splits. At the end, the three chunking methods enlarged.

Read the corpus slides

  1. Which passages in our corpus come from these three slides?

    Show the answer

    The Announcements slide gave the Homework 1 announcement and the project proposal date. The Project Team slide gave the team of 1-3 people, and these two facts make up the project page. The Chunking slide gave the full text of the syllabus. (Step 1)

  2. On the chunking slide, where do the fixed 50-token chunks cut the milestone date? What does that do to retrieval?

    Show the answer

    The first chunk ends at due Week 6, Thursday, and 11:59pm is in the second chunk. If the retriever returns only the first chunk, the reader does not get the full date. Where a passage should end is the subject of lesson 7. (Step 1)

3 points from this lesson

  1. Retrieval: for every query, pick a few passages from the corpus and put them into the prompt. It needs no training, and the answer comes with a citation.
  2. The retriever f(𝒟, Q) gives the top-K passages, and the reader g(Q, {P1, …, PK}) writes the answer from them. Word overlap gives the queue log and the old forum post the same score, 2, and bigrams do not break the tie either.
  3. A passage the retriever did not bring is never read, and that is the recall ceiling. In DrQA the ceiling is 77.8%, yet the whole system reaches 27.1%, because the ceiling is only an upper limit.

New terms in this lesson

Corpus (collection)
The set of all documents. For example, the company's 2,000 documents, or this course's 6 short texts.
Document
One item in the corpus: a web page, a file or an email. For example, the course syllabus.
Passage
A small piece of a document, which the retriever returns. For example, the syllabus paragraph that holds the milestone date.
Query
The question you search for. For example, “which office hour is least crowded”.
Retrieval
Finding useful passages for a query and putting them into the prompt. For example, the queue log going into the prompt with the question.
Citation
The source written next to an answer. For example, the [P3] in “Thursday 9am [P3]”, which points to the queue log.
Non-parametric memory
Knowledge kept outside the model, in the corpus. For example, the queue log: changing it needs no training.
Retriever
The part that scores every passage and returns the top K. For example, this lesson's word-overlap retriever.
Score
A number for how useful a passage is for the query. For example, the queue log's word-overlap score of 2.
Top-K
The K passages with the highest scores. For example, the queue log and the old forum post at K = 2.
Word overlap
How many of the query's words appear in the passage. The simplest score. For example, the queue log has office and hour, so its score is 2.
Stopwords
Very common words such as is, the and which, left out of the count. From “which office hour is least crowded”, which and is are removed.
Reader
The part that reads the query and the top-K passages and writes the answer. Today usually an LLM. For example, reading the queue log and writing “Thursday 9am”.
Zero-shot
Doing a task from an instruction alone, without training on that task. For example, the prompt line “Answer the question using the passage below”.
Recall
How often the retriever finds the passage that is needed. For example, 60 out of 100 questions gives a recall of 60%.
Recall ceiling
The retriever's recall becomes the upper limit of the whole system. With a recall of 60%, no reader can get more than 60 of 100 answers from the passages.
Reading comprehension
Answering a question about a text you are given. This is the reader's job after the retriever. For example, reading the Ottoman empire paragraph and answering 32.
SQuAD
Stanford Question Answering Dataset: questions people wrote about Wikipedia paragraphs, with the answer always a span of the paragraph. For example, 87,599 training questions.
Open-domain QA
A question on any topic that comes without its paragraph, so the system has to find it in the whole corpus. For example, DrQA's Warsaw question over all of Wikipedia.
DrQA
A 2017 retriever-reader system: a fixed TF-IDF retriever returns 5 Wikipedia articles, and a neural reader picks a span from them. For example, 833,500 for the Warsaw question.
Bigram
Two words that appear side by side. For example, office hour in the queue log, and least crowded in the forum post.
Distant supervision
Finding paragraphs automatically for data that has only questions and answers: run the retriever and keep the passages that contain the answer text exactly. For example, the syllabus for the milestone question.