CS 329Z · Engineering AI Agents · Lecture 3
RAG + Agents, from first principles
This lecture rests on one question: how do you give an LLM the right information at the right time, and where does that approach break? Here the question is split into 18 lessons, and all of them are ready. Each lesson builds on the earlier ones, and every new term is explained where it first appears.
How to read
- You can read the lessons in order, but you do not have to. The map below shows what each lesson needs first. No new term appears before its own lesson without an explanation. When it comes back in a later lesson, the name of its lesson is given there.
- Each lesson first says what the new idea is, then applies it to a real failure, such as BM25 ranking the old forum post above the queue log. At the top of each lesson is a ladder: under each step it says what you will be able to do after it. You can skip a step you already know.
- In many places there is a “Think first” question, and sometimes you work out a table row yourself. Solve it before opening the answer, with the arithmetic on paper.
- Change the numbers in the widgets. Each widget runs the same mechanism the lesson explains.
- Click any word with a dotted underline and ▾. Its definition opens right there, with a link to the lesson that explains it in full.
- Every lesson has two versions. The Easy version teaches the core idea step by step. The Dense version goes through every line of the Stanford slides, and also covers the paper details the slides rest on. Both exist in Hindi too, and the switch at the top of a lesson changes version at any time.
I assume you read code and know school maths, but that ML and statistics terms are new to you. If a lesson feels too easy or too heavy, say so.
Map of the lessons
This map shows which lesson rests on which. Each line is a dependency: the lower lesson needs the upper one. Lessons in the same row do not depend on each other, so you can read them in any order.
Click a lesson. The map shows what it needs first, what it opens up, and which lessons are independent of it.
The lines come from each lesson's “What you need first” line. A dependency that already follows from another path is not drawn again.
18 lessons
Each lesson starts from a gap left by an earlier one.
- 1How an LLM writes an answer, and where it gets stuckAsked about office hours, an LLM on its own confidently answered Wednesday without ever seeing the queue log. This lesson shows how an LLM builds an answer token by token, and where it gets stuck when the data is missing.LLM, Token, Vocabulary, Tokenizer, Probability, Next-token prediction, Parameters (weights), Training, Parametric knowledge, Knowledge cutoff, Prompt, Context window, HallucinationEasy · Dense
- 2Retrieval: finding before answeringAn LLM can only answer from what is in its prompt, and a prompt holds just 14 of a company's 60,000 passages. This lesson builds the simplest way to pick those 14 for every question.Corpus (collection), Document, Passage, Query, Retrieval, Citation, Non-parametric memory, Retriever, Score, Top-K, Word overlap, Stopwords, Reader, Zero-shot, Recall, Recall ceilingEasy · Dense
- 3Turning text into numbers: vectors and similarityThe query “when can I get help from a TA” shares no word with the office hours page that answers it, so word overlap gave every passage 0. Text has to be turned into vectors that can be compared.Vector, Dimension, Dot product (inner product), ∑ (sigma), Length, Cosine similarity, Normalize, Embedding, Embedding model (encoder), Index, Offline and onlineEasy · Dense
- 4Probability tools: getting ready to read the RAG formulaA retriever score of 1.56 and an LLM probability of 0.81 cannot simply be combined, yet lesson 5's RAG formula builds the answer's probability from exactly these numbers. This lesson builds the tools you need to run that formula by hand.Conditional probability, p(A|B), Chain rule (for probabilities), ∏ (capital pi), Marginalization, ≈, exp and e, Softmax, ∝, d⊤q, Natural log, ln, Loss, Likelihood, Negative log-likelihood, GradientEasy · Dense
- 5The RAG model: retriever and generator togetherAt the end of lesson 4 the retriever gave the old forum post more weight, and the right answer's probability fell from 0.685 to 0.315. The retriever has to be fixed, yet the training data holds only questions and answers.RAG, Latent variable, η and θ, BERT, MIPS, Generator, Encoder-decoder, Concatenate, RAG-Sequence, RAG-Token, End-to-end training, Backpropagation, Exact Match, Closed book and open bookEasy · Dense
- 6RAG pipeline: Naive, Advanced and ModularYou cannot train an LLM you rent through an API, so both mistakes from lesson 2 come back. They have to be fixed in the steps before the LLM.Frozen LLM, Pipeline, Naive RAG, Indexing time and query time, Pre-retrieval, Query routing, Query rewriting, Query expansion, Post-retrieval, Rerank and reranker, Summary, Fusion, Advanced RAG, Modular RAG, Pattern, Demonstrations (few-shot), Multi-hop questionEasy · Dense
- 7Chunking: cutting documents into pieces for retrievalCut the 40-page syllabus every 50 tokens and the milestone date splits across two chunks. Do not cut it at all, and the milestone's 2 lines dissolve into one vector for the whole syllabus.Word type and token in context, Static word embedding, Contextual embedding, Passage embedding, Multimodal embedding, Graph embedding, Mean pooling, Chunk, Fixed-size chunking, Overlap, Stride, Semantic chunking, Context-enriched chunking, Metadata, AI-driven dynamic chunkingEasy · Dense
- 8Giving a chunk back its lost contextAfter chunking, the ACME chunk has lost both the company name and the quarter, so the retriever cannot tell it apart from 49 near-identical chunks. The chunk needs its lost context back.Context loss, Contextual retrieval, Failed retrievals@20, Naive chunking, Late chunking, Long-context encoderEasy · Dense
- 9Questions spread across a whole document: RAPTOR and GraphRAGThe answer to “What are all the deadlines?” is spread over three passages, so top-K brings back only one piece of it. The index needs summaries of several passages, so that a question about the whole document finds its answer in one place.Detail question and thematic question, Clustering, Cluster and centroid, Soft clustering, RAPTOR, Tree, node, leaf, root, Entity, Relation, Knowledge graph and edge, Community, GraphRAG, Query-focused summarizationEasy · Dense
- 10Fast search over hundreds of millions of vectorsLesson 3's retriever compares the query with every vector in the index, and on 100 million passages each query takes about 7.7 seconds. The goal is to look at far fewer vectors and still find the nearest ones most of the time.Exact search (brute force), Big-O, O(N·d), Nearest neighbors, ANN, Distance, IVF, nprobe, Product quantization (PQ), Codebook, Proximity graph, Greedy search, HNSWEasy · Dense
- 11BM25: a score built from rare words, counts and lengthLesson 2's word overlap gave the queue log and the old forum post the same score of 2. Weighting each word by rarity, count and length breaks this tie, and the lesson shows which way it breaks.Term, Document frequency, n(q), IDF, TF-IDF, Term frequency, f(q, D), Saturation and k1, Length normalization, b and avgdl, BM25, Inverted index, Sparse and dense vectorEasy · Dense
- 12DPR: a retriever trained on question-passage pairsBM25 ranked the old forum post above the queue log. Two encoders, trained on question-passage pairs, have to bring the right passage to the top.DPR, Bi-encoder, Pretraining, Fine-tuning, Positive and negative passage, Contrastive loss, Batch, In-batch negatives, Hard negative, Top-k retrieval accuracyEasy · Dense
- 13Measuring a retriever: Hit@k, MRR, Recall@k and nDCGThe retriever improved and the second question's right passage moved to the top, yet the top-3 count stayed stuck at 0.667. We need numbers that can see that change.Relevance labels, Set, ∩, ∅, |A|, 𝟙[condition], Hit@k, Failed@k, MRR@k, Recall@k, log2, DCG and IDCG, nDCG@kEasy · Dense
- 14Hybrid search and rerankingBM25 put E2 at rank 3, and dense retrieval put it at rank 1. The two lists have to become one list, and then the top passages have to be read together with the query to fix their order.Paraphrase, Literal, Hybrid search, Reciprocal rank fusion (RRF), Attention, Cross-encoder, Reranking, ShortlistEasy · Dense
- 15ColBERT: a vector for every tokenFor the query error 4012, a one-vector bi-encoder ranked a short, wrong upload note above the long FAQ that has the answer. Keeping one vector per token, and adding up each query token's best match, brings the right passage to the top.Late interaction, MaxSim, ColBERT, Latency, FLOPs, CascadeEasy · Dense
- 16Agentic RAG: retrieval as a decisionBM25 ranked the old forum post above the queue log, and the pipeline sent it to the prompt without checking it. Retrieval has to become a decision at every step, and someone has to check what came back.Agent, Action and tool, Observation, Agentic RAG, Self-RAG, Reflection tokens, CRAG, Retrieval evaluator, Knowledge refinement, ThresholdEasy · Dense
- 17Search-R1: learning when and what to searchThe training data held only the right answer, not the search steps. The model has to learn when and what to search from the reward on its final answer alone.Reinforcement learning (RL), Policy, Rollout (trajectory), Reward, Search-R1, Advantage, PPO and value model, GRPO, Standard deviation, Reference LLM and KL penaltyEasy · Dense
- 18Where RAG breaks: 9 failures in retrieval, reading and the loopThe citation looked right, yet the answer was wrong, because the index still held the old office hours page. From outside every wrong answer looks the same, so the first job is to find where it went wrong: in retrieval, in reading, or in the agent's loop.Stale index, Permission leak, Lost in the middle, Distraction, Evidence conflict, Compounding error, No stopping rule, Prompt injectionEasy · Dense
Slides 2–8 are about the course project, so no lesson covers them.
All terms so far
The new terms from all 18 lessons, A to Z. Each term links to the lesson that explains it.
- ∑ (sigma)
- Keep adding over the range written below it. For example, the sum of aibi for i = 1 to 2 is a1b1 + a2b2.
- ∏ (capital pi)
- Multiply everything over the range written below it. For example, for three tokens, 0.21 × 0.31 × 0.71 = 0.046.
- ≈
- Approximately equal. The RAG formula adds up only the top-K passages, so the sum is only approximately equal.
- ∝
- Is proportional to. At the end, divide everything by one number so that the total is 1. For example, exps 2.718 and 1 become 0.731 and 0.269.
- η and θ
- The names of the retriever's (eta) and the generator's (theta) parameters. That is why the retriever is written pη(z|x) and the generator pθ(y|x, z).
- ⌈x⌉ (ceiling)
- The smallest whole number that is x or larger. For example, ⌈23.75⌉ = 24, and chunks = ⌈(1,000 − 50) ÷ 40⌉ + 1 = 25.
- ∂ (partial derivative)
- How fast an output changes when one input changes a little and everything else stays the same. For example, ∂𝓛/∂s+ = −0.814: raise the queue log's score by 0.01 and the loss falls by about 0.008.
- ℝd
- A list of d real numbers, that is, a vector with d dimensions. For example, [1.0, 0.5, 0.5, 1.0] ∈ ℝ4, and DPR's vectors are in ℝ768.
- ∇ (nabla)
- ∇θ𝓛: the loss's rate for every parameter in θ, as one list. Training nudges each parameter a little against its rate. For example, on the three scores these rates were −0.814, +0.506 and +0.307.
- 𝟙[condition]
- 1 if the condition is true, 0 if not. For example, q3's top 3 holds no right passage, so 𝟙[Rq3 ∩ Top3(q3) ≠ ∅] = 0.
- ⊆ (subset)
- A ⊆ B means everything in A is also in B. For example, Top3(q1) ⊆ Top5(q1), so Recall@5 is never lower than Recall@3.
- [CLS] and [SEP]
- Special tokens in BERT's input. [CLS] comes first, and in a cross-encoder its last vector carries the whole pair. [SEP] separates the query from the passage, as in [CLS] query [SEP] passage [SEP].
- Ablation
- Removing or freezing one part of a model and measuring again, to see what that part contributes. For example, freezing the query encoder in RAG-Sequence drops NQ (dev) from 44.0 to 41.2.
- Action and tool
- What the agent chooses to do, such as searching for
OH timings
. A tool is the program that does it, such as a search engine. - Advanced RAG
- The pipeline you get by adding pre-retrieval and post-retrieval steps to Naive RAG. For example, Rewrite, Retrieve, Rerank, Read.
- Advantage
- How much better or worse than expected a rollout did. For example, with rewards 0, 0, 0, 1, o4 gets +1.73 and the other three −0.58.
- Agent
- An LLM that works in a loop: it chooses an action, reads the observation, and decides the next action. For example, when the search for
OH timings
came back empty, the agent wrote a new query. - Agentic RAG
- RAG in which retrieval is an action of the agent, not a fixed step of a pipeline. For example, 0 searches for 17 × 3 and 3 for the Jed Hoyer question.
- AI-driven dynamic chunking
- Asking an LLM where the topic changes in a text, and cutting there. It needs LLM calls for every document.
- ANN
- Approximate nearest neighbor search: looking at fewer vectors and finding the right neighbors most of the time. For example, finding 9 of the true top-10 gives recall 0.9.
- Attention
- Each token takes a weighted average of the other tokens' vectors as its new vector, and the weights add up to 1. For example, if crowded [0.5, 0.5] gives weight 0.6 to averaged [1.0, 0.0], it becomes [0.8, 0.2].
- Backpropagation
- Working backwards from the loss to get every parameter's gradient, backprop for short. In RAG it reaches the generator and the query encoder, but not the passage encoder.
- BART
- A model that reads text and writes text, with about 400 million parameters. It is RAG's generator. For example, in the Jeopardy test BART on its own was judged more factual in only 7.1% of pairs.
- Batch
- A group of questions that training processes together. For example, a batch of 128 questions in DPR.
- Beam search
- Moving forward while keeping the B unfinished answers with the highest scores at each step. For example, with B = 2 the queue log (2.15) and forum post (1.475) answers go on, and Homework (0.425) is dropped.
- BERT
- An encoder model Google released in 2018. Both of the RAG retriever's encoders, the query encoder and the passage encoder, are built from it.
- Bi-encoder
- Two separate encoders for the question and the passage, with the score as their dot product. Passages are encoded in advance. For example, even with 21 million passages the model runs once per question.
- BIC
- ln(N) · k − 2 ln(L̂): choosing the number of clusters by weighing the fit's likelihood against its parameter count, where the smallest BIC wins. For example, on 6 passages K = 2 gives −23.64 and K = 3 gives −20.11.
- Big-O, O(N·d)
- How the cost grows as N or d grows. For example, in O(N·d), if N doubles, the cost doubles too.
- Bigram
- Two words that appear side by side. For example, office hour in the queue log, and least crowded in the forum post.
- BM25
- Each query term's IDF × its TF part, summed over the terms. For example, 2.749 for the forum post and 2.473 for the queue log.
- Candidate generation
- Picking candidate passages cheaply, so that the expensive arithmetic runs only on them. For example, in ColBERT each query vector brings back its top 1,000 token vectors, so at most 32,000 passages.
- Cascade
- A line of retrievers in which each later step does more precise, more expensive work on fewer passages. For example, from tens of millions to 1,000, then 50, then 5.
- Chain rule (for probabilities)
- p(A and B) = p(A) × p(B|A). With many steps, multiply them all. For example, p(Tuesday) × p(full | Tuesday) = 0.5 × 0.8 = 0.40. The chain rule for rates in lesson 12 is a different rule.
- Chain rule (for rates)
- Rates that are linked one after another multiply. For example, the rate of ln Z with respect to the queue log's score is 2.72 × (1 ÷ 14.59) = 0.186. Lesson 4's probability chain rule is a different rule.
- Chunk
- A piece of a document made for retrieval, and the basic unit of retrieval. For example, the Grading part of the 40-page syllabus.
- Citation
- The source written next to an answer. For example, the [P3] in “Thursday 9am [P3]”, which points to the queue log.
- Clipping and ε
- Keeping the ratio between 1 − ε and 1 + ε, then taking the smaller term with min. For example, with ε = 0.2 and A = +1.73, the term for a ratio of 1.5 stops at 2.076.
- Closed book and open book
- Answering from parameters alone, or searching documents first and then answering. For example, on NQ the closed-book T5-11B scores 34.5 and the open-book RAG-Sequence 44.5.
- Cluster and centroid
- A cluster is a group, and its centroid is the average of its vectors. For example, the centroid of [0.1, 0.9], [0.2, 0.8] and [0.15, 0.85] is [0.15, 0.85].
- Clustering
- Splitting vectors into groups so that the vectors in a group are close to each other. For example, the syllabus [0.1, 0.9] and the project page [0.2, 0.8] have cosine 0.99, so they share a group.
- Codebook
- The list of sample vectors (codewords) for a piece. For example, #0 [1, 0], #1 [0, 1], #2 [0.5, 0.5], #3 [0.7, 0.7].
- ColBERT
- A late-interaction retriever that keeps a vector per token and scores with MaxSim. Built by Khattab and Zaharia in 2020. For example, its Recall@1000 on MS MARCO is 96.8.
- Column vector and ⊤
- A column vector has its numbers written one above the other, and ⊤ (transpose) turns it into a row. A row times a column is the dot product. For example, the row [0.9, 0.1] times the column [0.8, 0.2] = 0.74.
- Community
- A tightly linked part of a graph. For example, Office hours, TAs, Room 104 and Zoom.
- Compounding error
- An early mistake carries into every later step. For example, with each step right at 0.9, 3 steps give 0.729 and 5 steps give 0.59.
- Concatenate
- Join two texts end to end into one text. For example,
question: …
followed bypassage: …
. - Conditional probability, p(A|B)
- The probability of A when B is known. Read “|” as “given”. For example, p(full | Tuesday) = 16 ÷ 20 = 0.80.
- Context loss
- After cutting, a chunk can no longer say which document or topic it belongs to. For example,
The company’s revenue grew by 3%
does not name the company. - Context window
- The most tokens a model can read at once. Both the prompt and the answer count. For example, an 8,000-token window with a 7,500-token prompt leaves 500 for the answer.
- Context-enriched chunking
- Storing metadata or a summary with every chunk. For example,
Source: CS 329Z syllabus, section: Grading
above every chunk. - Contextual embedding
- A separate vector for each token, shaped by its sentence. For example, BERT. In
Late days may not be applied
late is [0.1, 0.9], and inlate afternoon office hours
it is [0.9, 0.1]. my numbers - Contextual retrieval
- Indexing each chunk with a 50–100 token context, written by an LLM, placed in front of it. For example, adding
From ACME Corp’s Q2 2023 SEC filing
. - Contrastive loss
- Softmax over the positive and negative scores, then −ln of the positive's share. It looks only at differences between scores. For example, scores 1.0, 2.0, 1.5 give loss 1.68, and 5.0, 2.0, 0.5 give 0.059.
- Corpus (collection)
- The set of all documents. For example, the company's 2,000 documents, or this course's 6 short texts.
- Cosine similarity
- The dot product divided by the product of both lengths. It looks only at direction and stays between −1 and 1. For example, 0.74 ÷ (0.82 × 0.91) = 0.99.
- CRAG
- Corrective RAG: an evaluator calls the retrieval Correct, Incorrect or Ambiguous, and refinement, web search or both follow. For example, the forum post at 0.4 is Ambiguous.
- Critic
- A model that grades what something else wrote or retrieved. For example, in Self-RAG a critic built from Llama 2 7B adds reflection tokens to the training data, and the CRAG slide calls its evaluator a critic.
- Cross-encoder
- A reranker that joins the query and the passage, feeds them to one model, and gets a direct score for the pair. It runs in full for every passage, for example 10.7 ms per passage with BERT-base.
- DCG and IDCG
- Each rank's grade ÷ log2(rank + 1), added up. The IDCG is the DCG of the best order. For example, q1's order queue log (3), forum post (0), office hours page (1) gives DCG 3.5 and IDCG 3.631.
- Decomposable similarity
- A score that comes from a question vector and a passage vector, where the passage vector is built without seeing the question. Only then can an offline index exist. For example, the dot product, cosine and distance.
- Demonstrations (few-shot)
- Worked examples placed in the prompt before the question. For example, “Q: When is the milestone due? A: Week 6, Thursday [P1]”.
- Detail question and thematic question
- The answer is written in one place, or it is built from several passages. For example, “When is the milestone due?” and “What are all the deadlines?”.
- Dimension
- How many numbers a vector has. For example, [0.9, 0.1] has dimension 2, and BERT-base vectors have 768.
- Distance
- The length of the difference between two vectors. For example, [0.8, 0.2] and [0.9, 0.1] differ by [0.1, −0.1], so the distance is √(0.01 + 0.01) = 0.14.
- Distant supervision
- Finding paragraphs automatically for data that has only questions and answers: run the retriever and keep the passages that contain the answer text exactly. For example, the syllabus for the milestone question.
- Distraction
- A similar but useless passage leads the reader astray. For example, the old 2024 forum post.
- Document
- One item in the corpus: a web page, a file or an email. For example, the course syllabus.
- Document frequency, n(q)
- How many documents contain term q. For example, n = 1 for milestone and 2 for office.
- Dot product (inner product)
- Multiply the numbers in the same position, then add them all. For example, [0.8, 0.2] and [0.9, 0.1]: 0.72 + 0.02 = 0.74.
- DPR
- Dense Passage Retrieval: a bi-encoder retriever with two BERT encoders, trained with contrastive loss on question-passage pairs. Built by Karpukhin and colleagues in 2020, and the retriever inside lesson 5's RAG.
- DrQA
- A 2017 retriever-reader system: a fixed TF-IDF retriever returns 5 Wikipedia articles, and a neural reader picks a span from them. For example, 833,500 for the Warsaw question.
- DSP
- Demonstrate–Search–Predict: a program that passes text between a frozen LM and a frozen retriever, and builds its own demonstrations. Built at Stanford by Khattab and colleagues in 2022. For example, Exact Match 51.4 on HotPotQA.
- d⊤q
- A way of writing the dot product of vectors d and q. For example, the office hours page [0.9, 0.1] and the question [0.8, 0.2] give 0.74.
- ef
- The size of the list of nearest nodes found so far in an HNSW search. A larger ef means more distances and higher recall. For example, starting at node 70 with ef = 3 gives {75, 70, 80}.
- Embedding
- A text's vector, produced by a trained model. Texts saying similar things point in similar directions, such as “help from a TA” and “run by the TAs”.
- Embedding model (encoder)
- A model that reads a text and produces its fixed-length embedding. For example, turning the query into a vector like [0.8, 0.2].
- Encoder-decoder
- The encoder turns the input into vectors, and the decoder looks at them while it writes one token at a time. BART is such a model.
- End-to-end training
- Training all the parts together with one loss. In RAG the query encoder and the generator both learn from the same −ln p(y|x). For example, −ln 0.70 = 0.357.
- Entity
- Any named thing: a person, a place, a course, a document or a date. For example, Week 6, TAs, Room 104.
- Evidence conflict
- Two sources, or a source and the model's parameters, say different things. For example, Week 6 and Week 7.
- Exact Match
- An answer counts as right only if it matches the correct answer exactly. The score is the percentage of such questions. For example, RAG-Sequence's 44.5 on NQ.
- Exact search (brute force)
- Comparing with every vector to get the guaranteed top-k. For example, with 21 vectors, computing 21 distances.
- exp and e
- exp(s) = es, where e is about 2.718. Always above 0, and ea+1 = ea × e. For example, exp(1) = 2.718, exp(0) = 1, exp(−1) = 0.368.
- Expected value, 𝔼[ · ]
- Multiplying each outcome by its probability and adding, that is, the average over many rollouts. For example, 0.6 × 1 + 0.4 × 0 = 0.6.
- F1
- A score that combines precision and recall, 2PR ÷ (P + R). For example, P = 0.75 and R = 0.6 give F1 = 0.667.
- Failed retrievals@20
- When each query has one right chunk, the percentage of queries whose right chunk is not in the top 20. Smaller is better. For example, 29 failures in 1,000 queries is 2.9%.
- Failed@k
- 1 − Hit@k, the share of queries with no right passage in the top k. For example, Failed@3 = 1 − 0.667 = 0.333.
- FAISS
- An open-source library for fast search over vectors. For example, DPR built an HNSW index over 21 million vectors in it, which took about 8.5 hours.
- Fine-tuning
- Training an already trained model further on data for your own task. For example, DPR trains BERT on question-passage pairs.
- Fixed-size chunking
- Cutting at a fixed number of tokens, even if a fact breaks in the middle. For example, every 50 tokens.
- FLOPs
- How many multiplications and additions it took, that is, the computer's work. For example, a dot product of 3 numbers takes 6 FLOPs, and one of 768 numbers takes 1,536.
- Frozen LLM
- An LLM whose parameters you cannot change. You can only send it prompts. For example, GPT or Gemini used through an API.
- Fusion
- Merging several ranked lists into one. For example, one list is queue log, forum post, office hours page, and another is office hours page, queue log. The passages that appear in both rise to the top.
- GAE
- Generalized advantage estimation: at every step δ = reward + next estimate − this estimate, then the sum of all later δ. With γ = λ = 1 the advantage is reward − V, such as 1 − 0.3 = 0.7.
- Gaussian mixture model (GMM)
- A weighted sum of bell curves: P(x) = ∑ πk 𝒩k(x). It gives every passage a membership for every cluster. For example, [0.6, 0.4] gets 0.904 for office hours and 0.096 for deadlines.
- Generator
- The model that reads the question and one passage and writes the answer, pθ(y|x, z). In RAG it is BART. For example, 0.9 for
Thursday 9am
with the queue log. - GiB
- 230 = 1,073,741,824 bytes, about 7% more than a GB. For example, ColBERT's MS MARCO index is 286 GiB.
- Gradient
- The calculation of which way, and how much, to change each parameter so that the loss goes down. For example, in lesson 1 training moved the parameters in the direction that raised Thursday's probability from 0.10 to 0.12 and lowered the loss from 2.303 to 2.120.
- Graph embedding
- A vector for a node or a relation in a network. For example, node2vec. In a course graph: the TA, office hours, and the runs relation between them.
- GraphRAG
- Answering global questions from a knowledge graph, its communities and their summaries. For example, the all-deadlines answer, built from the Project and Homework 1 communities.
- Greedy decoding and sampling
- Greedy decoding takes the highest-probability token at every step, and sampling chooses at random according to the probabilities. For example, greedy always writes Wednesday afternoon, while sampling starts with Thursday about 21 times in 100.
- Greedy search
- Moving at every step to the neighbor nearest to the query, until stuck. For example, moving from 70 to 75 for the query 73.
- GRPO
- An advantage from the mean and standard deviation of a group's rewards. No value model is needed. For example, for 0, 0, 0, 1 the mean is 0.25 and the standard deviation 0.433.
- Hallucination
- A confident answer that is wrong or has no source. For example, Wednesday afternoon, written without seeing the log.
- Hard negative
- A wrong passage that looks similar to the question. DPR takes these from BM25's top passages. For example, the forum post for our question.
- Harmonic series
- 1 + 1/2 + 1/3 + …, the 1 ÷ rank numbers that MRR uses. Their sum grows slowly, about ln n + 0.577. For example, the first 1,000 add up to 7.485.
- Hit@k
- The share of queries with at least one right passage in the top k. For example, at k = 3, 2 of 3 queries, so 0.667.
- HNSW
- A layered proximity graph: a greedy walk from the top, dropping one layer when stuck. For example, 4 steps for the query 73.
- Hybrid search
- Running both a word-based and a dense retriever and merging their lists. For example, making one list from BM25's E1, E3, E2 and dense retrieval's E2, E1, E3.
- Hyperparameter
- A number in a formula or model that training does not change, picked by trying values on a dev set. For example, BM25's k1 and b, which the DPR paper set to 0.9 and 0.4.
- i.i.d. and conditional embeddings
- i.i.d.: each chunk’s vector is built completely separately from the other chunks, as in naive chunking. Conditional: each chunk’s vector is built with the whole document in view, as in late chunking, where
It
gets [0.9, 0.1]. - IDF
- ln((N − n + 0.5) ÷ (n + 0.5)), a bigger weight for a rarer term. For example, 1.299 for milestone and 0.588 for office.
- In-batch negatives
- Using the positives of the other questions in the batch as negatives. For example, in a batch of 128 each question gets 127 negatives.
- Increasing function
- A function where a larger input always gives a larger output, so the order is kept. exp and ln are such functions. For example, scores 1.56 and 0.74 give weights 0.499 and 0.220, in the same order.
- Incremental re-indexing
- Making embeddings only for new and changed passages, and removing the old passages of a changed document, instead of rebuilding the whole index. For example, 500 + 30 = 530 embeddings instead of 60,000.
- Index
- The list of passage embeddings built in advance. For example, 60,000 vectors for 60,000 passages.
- Index hot-swapping
- Replacing the document index without touching the parameters, so that the model's knowledge changes. For example, the 2018 index answered 68% of questions about 2018 leaders correctly, and the 2016 index only 4%.
- Indexing time and query time
- Steps that run once, and steps that run for every query. For example, the embeddings of 60,000 passages are made once, and the query's embedding every time.
- Interaction matrix
- A table of the similarity of every query token with every passage token. For example, the FAQ's table holds 6 × 2 = 12 dot products.
- Inverted index
- For each term, the list of documents that contain it. For example, office → office hours page, queue log.
- ITER-RETGEN
- Retrieval and generation in turns, where the previous answer joined to the question becomes the next query. Built by Shao and colleagues in 2023. For example, adding Thursday 9am lifts the office hours page from 1 to 3.
- Iterative, recursive and adaptive retrieval
- Three ways to search more than once: in turns for a fixed count, by breaking the question into smaller ones, and by letting the model decide. For example, ITER-RETGEN is iterative, and the Search-R1 trace is adaptive.
- IVF
- Inverted file index: splitting the vectors into clusters and looking only at the clusters near the query. For example, 11,000 comparisons for 1 million vectors.
- k-means
- A way to split vectors into k clusters: give each vector to its nearest centroid, then make each centroid the average, and repeat. For example, on 6 passages it ends after 3 rounds with A = [0.817, 0.2] and B = [0.2, 0.8].
- KL divergence
- The average of ln(new probability ÷ reference probability), weighted by the new model's probabilities. For example, 1.146 for 0.2 and 0.8 becoming 0.9 and 0.1, and 0.007 for 0.25 and 0.75.
- Knowledge cutoff
- The date on which training stopped. Later events, and your private data, are not in the model. For example, with a March 2026 cutoff, an October 2026 announcement is not in the model.
- Knowledge graph and edge
- A network of entities (nodes) and relations (edges). For example, an edge labelled “due” between the milestone and Week 6.
- Knowledge refinement
- Splitting documents into strips, dropping the useless strips, and joining the rest in order. For example, 2 of the syllabus's 3 strips stay for the milestone question.
- Late chunking
- Encoding the whole document first, then mean pooling each chunk’s token vectors. Then the vector of
It is due Week 6
becomes [0.9, 0.1]. - Late interaction
- The query's and the passage's token vectors are built separately, and meet only at scoring. For example, ColBERT builds passage vectors offline, and computes the dot products when the query arrives.
- Latency
- The time from sending a query to getting the answer, in ms. For example, the cross-encoder's latency is 10,700 ms for 1,000 passages, and ColBERT's reranking latency is 61 ms.
- Latent variable
- A hidden choice that the data does not record. In RAG, the choice of which passage the answer came from. For example, a question-answer pair does not say that the answer came from the queue log.
- Leiden and degree
- Leiden splits a graph into communities with many edges inside. A node's degree is its number of edges. For example, Project has degree 3.
- Length
- The length of the arrow: square each number, add the squares, take the square root. For example, [0.8, 0.2] has length √(0.64 + 0.04) = 0.82.
- Length normalization, b and avgdl
- Lowering the TF part of documents longer than average, by an amount set by b. For example, with avgdl = 16.17 and b = 0.75, a document twice the average length gets 0.710.
- Likelihood
- The probability the model gave the correct data. For example, if it gave the right answer 0.12, the likelihood is 0.12.
- Literal
- Text that has to match exactly, such as the error code E-4012 or the order ID A77-310. A dense retriever can miss it, and BM25 catches it as a rare word.
- LLM
- Large language model: a model that reads text and writes what comes next, one token at a time. For example, the model that wrote Wednesday afternoon.
- log2
- The power you must raise 2 to in order to get a number. For example, log2 4 = 2, and log2 3 = 1.585.
- Long late chunking
- Late chunking for a document longer than the encoder’s limit, read in large overlapping windows. For example, 20,000 tokens read in windows of 8,192 tokens, each new window starting 1,000 tokens back.
- Long-context encoder
- An encoder that can read a very long text in one go. For example, 8,192 tokens, about 10 pages.
- Lookup table (ADC)
- Computing the score of every query piece with every codeword in advance and keeping them in a table. Each passage's score is then just a sum of table entries. For example, the score of code (0, 1) is 1 + 1 = 2.0.
- Loss
- A number for how wrong the model is. Training makes it small. For example, if the right answer has probability 0.5, the loss is −ln 0.5 = 0.693.
- Lost in the middle
- An LLM uses the middle of a long prompt less (position bias). For example, with 20 passages, 75.8% right at first place and 53.8% at 10th.
- Map-reduce
- Doing the same job on each piece separately (map), then combining the results (reduce). For example, a partial answer and a score from each community, then a final answer in score order.
- Marginalization
- Each case's weight times the probability within that case, added up over all cases. For example, 0.5 × 0.8 + 0.5 × 0.1 = 0.45.
- Matrix and matrix-vector product
- A table of numbers in rows and columns, and the dot product of each of its rows with a vector. For example, our 4 × 2 index times the query [0.8, 0.2] = [0.74, 0.26, 0.32, 1.56].
- MaxSim
- Each query token's largest dot product with the passage's tokens, then the sum of these. For example, 0.9 + 0.95 = 1.85 for the course FAQ.
- Mean pooling
- Making one vector by averaging several token vectors. For example, the average of [0.9, 0.1], [0.7, 0.3] and [0.8, 0.2] is [0.8, 0.2].
- Metadata
- Data about data: name, section, date. For example,
CS 329Z syllabus, Grading, 30 Sep 2026
. - Min-max normalisation
- Mapping each list's scores onto 0 to 1 with (x − min) ÷ (max − min). For example, BM25's 7.2, 3.1 and 1.0 become 1, 0.339 and 0. The result depends on the list's min and max.
- MIPS
- Maximum inner product search: finding the K passages in the index with the largest dot products. For example, the 5 highest scores among 21 million passages.
- Modular RAG
- Splitting the pipeline into modules and connecting them in different orders. For example, Retrieve, Read, Retrieve, Read.
- MRR@k
- 1 ÷ the rank of the first right passage, averaged over the queries, with 0 if none is in the top k. For example, ranks 1, 2 and none give (1 + 0.5 + 0) ÷ 3 = 0.5.
- Multi-hop question
- A question that needs several searches, because the parts of the answer sit in different passages. For example, where the least crowded office hour is: first the queue log, then the office hours page.
- Multimodal embedding
- Vectors for text and for images in the same space. For example, CLIP. A photo of office hours written on a whiteboard and the text
office hours schedule
can get nearby vectors. - Naive chunking
- Cutting first, then encoding each chunk on its own. Then the vector of
It is due Week 6
stays at [0.5, 0.5]. - Naive RAG
- The simplest pipeline: an index, then Retrieve and Read for every query, with no extra step in between.
- Natural log, ln
- The power you raise e to in order to get x. ln(ex) = x, and ln(a × b) = ln a + ln b. For example, ln(0.1) = −2.303.
- nDCG@k
- DCG ÷ IDCG, between 0 and 1, which looks at both grade and order. For example, q1's reversed order gives 2.5 ÷ 3.631 = 0.689.
- Nearest neighbors
- The vectors nearest to the query. For example, among the neighbors of the query 73, 75 comes first.
- Negative log-likelihood
- −ln p(right answer). The most common loss. For example, at p = 0.12, −ln 0.12 = 2.120.
- Negative marginal log-likelihood
- −ln of the right answer's probability, when that probability comes from marginalization over passages, added up over all examples. RAG's training loss. For example, over two examples 0.378 + 0.693 = 1.071.
- Next-token prediction
- Looking at the text so far, giving every token a probability, picking one, adding it, and repeating. For example, Wednesday, then afternoon, then a full stop.
- No stopping rule
- The agent has no clear condition for stopping. For example, 15 searches on one question.
- Non-parametric memory
- Knowledge kept outside the model, in the corpus. For example, the queue log: changing it needs no training.
- Normalize
- Divide every number in a vector by its length, so the length becomes 1 and the direction stays the same. For example, [0.8, 0.2] ÷ 0.82 = [0.97, 0.24].
- Normalizer, Z
- In softmax, the sum of all the exps. Dividing by it makes the weights add up to 1, and it is the division hidden behind ∝. For example, for exps 2.718 and 1, Z = 3.718.
- nprobe
- How many clusters IVF looks at. For example, with 1,000 clusters and nprobe = 10, only the vectors of 10 clusters are checked.
- Observation
- The result of an action, which the agent reads before its next step. For example, the result of the first search: every passage scored 0.
- Offline and online
- Offline: once, before any query arrives, such as building the passage embeddings. Online: on every query, such as making the query's embedding.
- Open-domain QA
- A question on any topic that comes without its paragraph, so the system has to find it in the whole corpus. For example, DrQA's Warsaw question over all of Wikipedia.
- ORQA
- A 2019 dense retriever that trained its retriever with extra pretraining. For example, on NQ the full system's Exact Match is 33.3 with ORQA and 41.5 with DPR.
- Overlap
- A new chunk starts by repeating the last few tokens of the previous one. To keep a fact of L tokens whole, it must be at least L − 1. For example, 6 for a 7-word date.
- Parameters (weights)
- The numbers inside a model that produce the probability of every next token. For example, T5-11B has about 11 billion parameters.
- Parametric knowledge
- Knowledge spread across the parameters. The model cannot name its source, and a new fact needs training again.
- Paraphrase
- The same statement in other words. For example, upload failed and upload keeps failing. BM25 misses it, because it counts shared words.
- Passage
- A small piece of a document, which the retriever returns. For example, the syllabus paragraph that holds the milestone date.
- Passage embedding
- One vector for a whole chunk, the kind the retrieval index stores. For example, the vectors from Sentence-BERT, DPR and E5.
- Pattern
- A fixed order of modules, such as Retrieve, Read.
- Permission leak
- The retriever brings a passage this user is not allowed to see. For example, other students' grades for a student.
- Pipeline
- A line of steps in which the output of each step is the input of the next. For example, Retrieve outputs the top-K passages, and those are the input of Read.
- Points (absolute difference)
- The plain difference between two percentages. For example, 52.2 to 54.0 is 1.8 points, which is an improvement of about 3.45% relative to 52.2.
- Policy
- Whatever makes the decision at each step. Here the LLM, which picks the next token every time: think, <search> or <answer>.
- Positive and negative passage
- The question's correct passage (pi+), and wrong passages (pi,j−). For example, the least-crowded question's positive is the queue log, and its negatives are the forum post and the Homework announcement.
- Post-retrieval
- Steps that improve the returned list after retrieval: rerank, summary and fusion.
- Posterior and prior
- The prior pη(z|x) is a passage's weight before seeing the answer, and the posterior p(z|x, y) is its weight after, that passage's share of the total probability. For example, the queue log's prior is 0.602 and its posterior 0.773.
- PPO and value model
- A common RL method that trains a separate model (the value model) to estimate the coming reward. For example, an estimate of 0.25 and a reward of 1 give an advantage of +0.75.
- Pre-filter and post-filter
- A pre-filter removes forbidden passages before ranking, and a post-filter after taking the top K. For example, with 5 forbidden passages in the top 14, a post-filter leaves 9.
- Pre-retrieval
- Steps that improve the query before retrieval: routing, rewriting and expansion.
- Precision
- The share of the returned passages that are right. For example, at K = 2 the queue log is right and the forum post is wrong, so precision = 1 ÷ 2 = 0.5.
- Pretraining
- A model's first, large training on lots of general text. For example, BERT is first trained on general text.
- Primacy and recency bias
- An LLM's habit of using the start (primacy) and the end (recency) of a prompt more. For example, with 20 passages, 75.8% at the first position, 63.2% at the last and 53.8% at the 10th.
- Probability
- A number between 0 and 1 that gives the chance of something. The options always add up to 1. For example, a quiz in 5 of 20 lectures gives 5 ÷ 20 = 0.25.
- Probability ratio
- πθ(token) ÷ πold(token), how many times larger a token's probability is than under the model that made the rollout. For example, 0.55 ÷ 0.5 = 1.1.
- Product quantization (PQ)
- Cutting a vector into pieces and storing, for each piece, the number of its nearest codeword. For example, [0.9, 0.1, 0.2, 0.8] becomes the code (0, 1).
- Prompt
- The text you give the model right now: the question, instructions, and accompanying documents. For example, the queue log line that lifted Thursday from 0.21 to 0.81.
- Prompt caching
- The LLM service keeps a repeated part of the prompt after the first call, such as the whole document, so that part is cheap in later calls. According to the blog, contextual retrieval then costs $1.02 per million document tokens.
- Prompt injection
- Retrieved text becomes a command for the agent. For example, a hidden
Ignore previous instructions
in a post. - Proximity graph
- A graph in which every vector is linked to its nearby vectors. For example, 70 is linked to its neighbors 65 and 75.
- QRels
- A file of relevance labels: the IDs of the right documents for each query. For example, the late chunking paper computed nDCG@10 from such a file.
- Query
- The question you search for. For example, “which office hour is least crowded”.
- Query augmentation
- Filling the query with [mask] tokens up to 32 tokens. BERT builds vectors for these too, and they join MaxSim like extra query tokens.
- Query expansion
- Adding related words or other forms to the query. For example, adding “office hours” to “help ta” raises the office hours page from 0 to 2.
- Query rewriting
- Having an LLM turn the query into a clean search query. For example, “OH timings?” becomes “CS 329Z office hours schedule”.
- Query routing
- Deciding which index or source to search. For example, “when is the project milestone due” goes to the syllabus index.
- Query-focused summarization
- Writing a summary with the question in mind. For example, asking for “all the deadlines” makes the Project community's summary give only the dates.
- RAG
- Retrieval-Augmented Generation: joining a retriever and a generator into one model and training them together with one loss. For example, bringing the queue log and writing
Thursday 9am
from it. - RAG-Sequence
- The whole answer with each passage, then the weighted sum. For example, 0.6 × 0.18 + 0.4 × 0.45 = 0.288.
- RAG-Token
- A weighted sum at every token, then the product of the tokens. For example, 0.74 × 0.48 = 0.355.
- RAPTOR
- Building a tree bottom up from summaries of clusters, and searching every level. For example, the all-deadlines question matches summary A.
- Reader
- The part that reads the query and the top-K passages and writes the answer. Today usually an LLM. For example, reading the queue log and writing “Thursday 9am”.
- Reading comprehension
- Answering a question about a text you are given. This is the reader's job after the retriever. For example, reading the Ottoman empire paragraph and answering 32.
- Recall
- How often the retriever finds the passage that is needed. For example, 60 out of 100 questions gives a recall of 60%.
- Recall ceiling
- The retriever's recall becomes the upper limit of the whole system. With a recall of 60%, no reader can get more than 60 of 100 answers from the passages.
- Recall@k
- Right passages found in the top k ÷ all right passages, averaged over the queries. For example, at k = 1 q1 found 1 of 2, so 0.5.
- Reciprocal rank fusion (RRF)
- 1 ÷ (k + rank) from a passage's rank in each list, summed over all lists, with k = 60. For example, E1: 1 ÷ 61 + 1 ÷ 62 = 0.03252.
- Recursive chunking
- Cutting along a ladder of separators: paragraph first, then line, then sentence, then space. Only a piece still larger than the chunk size gets cut at the next separator. For example, at size 20 the Grading section becomes two chunks of 19 and 7 words.
- Reference LLM and KL penalty
- A frozen copy of the model from before training, and a penalty that stops the new model from moving far from it. For example, if <search> goes from probability 0.2 to 0.9, the penalty is large.
- Reflection tokens
- Special tokens added to the vocabulary, which the model writes like any other text. For example, Retrieve, Relevant or Irrelevant, and Supported or Partially.
- Reinforcement learning (RL)
- Learning by making attempts and using each attempt's reward. For example, if 2 of 4 rollouts have reward 1, the probability of the tokens in those two goes up.
- Rejection sampling
- Making several rollouts per question and fine-tuning only on those with the right answer. For example, 5 rollouts per question in the paper, with an average of 0.348 over 7 datasets.
- Relation
- A link between two entities. For example, (Project milestone, due, Week 6).
- Relative gain
- The difference divided by the old number, written as a percentage. For example, (51.4 − 36.9) ÷ 36.9 = 0.393, about 39%. The plain difference, 14.5, is in percentage points.
- Relevance labels
- The right passages fixed in advance for each query, sometimes with grades. For example, for q1 the queue log (grade 3) and the office hours page (grade 1).
- Rerank and reranker
- A slower but more accurate model that sorts the returned passages again. For example, it puts the forum post below the queue log when both tie at 2.
- Reranking
- Scoring and sorting a fast retriever's shortlist again with an accurate model. For example, on MS MARCO, MRR@10 goes from 16.7 for BM25 alone to 34.7 after reranking with BERT-base.
- Residual
- What is left after subtracting a vector's cluster centroid from it. IVFADC stores the code of this difference. For example, queue log [0.85, 0.2] − [0.817, 0.2] = [0.033, 0].
- Retrieval
- Finding useful passages for a query and putting them into the prompt. For example, the queue log going into the prompt with the question.
- Retrieval collapse
- The retriever starts bringing the same passages for every question during training, after which the generator learns to ignore them. For example, on the paper's story-writing task.
- Retrieval evaluator
- A small model that reads the question and one document and returns a confidence. For example, 0.4 for the forum post at K = 1.
- Retrieved token masking
- Computing training terms only on tokens the model wrote, leaving out the tokens inside <information>. For example, 114 of the 430 words in the slide's trace. Without the mask the average falls from 0.431 to 0.343.
- Retriever
- The part that scores every passage and returns the top K. For example, this lesson's word-overlap retriever.
- Reward
- The number at the end of a rollout. For example, 1 if the answer matches exactly, 0 otherwise.
- Rollout (trajectory)
- One complete attempt at a question, from the start to the final answer. For example, o1 on the Hoyer question: it searched both dates, then answered.
- Rule for choosing neighbors (heuristic)
- A candidate is linked only if it is nearer to the new node than to every node already linked. For example, X links to A and E2, and B and C are skipped.
- Salient span masking (SSM)
- An extra round of pretraining in which the model fills in blanked-out names and dates in Wikipedia sentences. For example, it lifted T5-11B's TriviaQA score from 50.1 to 60.5.
- Saturation and k1
- As the count grows, the gain fills up slowly towards a ceiling of k1 + 1. For example, with k1 = 1.2 the TF part for f = 1, 2, 10 is 1.000, 1.375, 1.964.
- Score
- A number for how useful a passage is for the query. For example, the queue log's word-overlap score of 2.
- Search-R1
- A method that teaches an LLM search decisions with RL, using think, search, information and answer tags. The reward is given only on the final answer.
- Segment score (Self-RAG)
- The weighted sum of a segment's critique shares: S = 1.0 × sIsRel + 1.0 × sIsSup + 0.5 × sIsUse. For example, the queue log segment's S = 0.9 + 0.875 + 0.5 × 0.75 = 2.15.
- Self-RAG
- The generator itself writes reflection tokens to decide when to retrieve and which segment to keep. For example, Relevant and Partially on the forum post segment.
- Semantic chunking
- Cutting at logical boundaries such as sentences or sections. For example, cutting the syllabus at its Grading, Office hours and Project headings.
- Set, ∩, ∅, |A|
- A group of things, the things two sets share, the empty set, and how many things a set holds. For example, |{queue log, office hours page}| = 2, and for q3, Rq3 ∩ Top3(q3) = ∅.
- Shortlist
- The retriever's top few passages, on which the reranker runs. For example, the top 100 of 60,000 passages, which the cross-encoder handles in 1,070 ms.
- Soft clustering
- Putting one passage into more than one cluster. For example, the FAQ page, whose cosine is high with both centroids (0.86 and 0.78).
- Softmax
- The exp of each score, divided by the sum of all the exps. It turns scores into probabilities. For example, 2.0, 1.0, 0.1 become 0.659, 0.242, 0.099.
- Sparse and dense vector
- A long vector that is mostly zeros, and a short one that is almost full. For example, 55 of the 69 slots in the forum post's vector are 0.
- SQuAD
- Stanford Question Answering Dataset: questions people wrote about Wikipedia paragraphs, with the answer always a span of the paragraph. For example, 87,599 training questions.
- Stale index
- The document changed, but the index keeps its old passage and embedding. For example, Tuesday 4pm after the move to Wednesday 2pm.
- Standard deviation
- A measure of how spread out numbers are: the average of the squared differences from the mean, then the square root. For example, for 0, 0, 0, 1 the mean is 0.25, the squares are 0.5625, 0.0625, 0.0625, 0.0625, their average is 0.1875, and the square root is 0.433.
- Static word embedding
- One vector per word type, whatever the context. For example, word2vec and GloVe. The vector for late stays [0.5, 0.5] in both sentences. my numbers
- Stopwords
- Very common words such as is, the and which, left out of the count. From “which office hour is least crowded”, which and is are removed.
- Stride
- The distance between the starts of two neighbouring chunks: chunk size − overlap. For example, 9 − 6 = 3.
- Summary
- Shortening passages before the prompt. For example, if a 500-token passage becomes 100 tokens, 14 passages take 1,400 tokens instead of 7,000. The risk: a detail such as Room 104 can drop out.
- Term
- One word of the query in search. For example, office and hour in
which office hour
. - Term frequency, f(q, D)
- How many times term q appears in document D. For example, office has f = 2 in the queue log.
- TF-IDF
- The term's count × ln(N ÷ n), an older method than BM25. For example, N = 6, n = 2, f = 2 gives 2 × ln 3 = 2.20.
- Thorough and Fast decoding
- Two ways for RAG-Sequence to write an answer. Each passage produces one candidate. Thorough checks every candidate with every passage, Fast only with the passage that produced it. For example, with 50 passages on NQ, Thorough needs up to about 2,500 generator runs.
- Threshold
- A limit set in advance, where being above or below it triggers a different action. For example, in CRAG above 0.6 means Correct, and below 0.2 means Incorrect.
- Token
- A small piece of text: a word, part of a word, or punctuation. The model reads and writes these. For example, crowded is two tokens, crowd and ed.
- Tokenizer
- The program that splits text into the vocabulary's tokens. For example, it splits Wednesdayy into Wednesday and y.
- Top-K
- The K passages with the highest scores. For example, the queue log and the old forum post at K = 2.
- Top-k retrieval accuracy
- The percentage of questions with an answer passage in the top k. For example, DPR's 78.4% on NQ at top-20.
- Training
- Changing the parameters little by little so that the real next token in the text gets a higher probability. For example, Thursday after Office hours are on goes from 0.10 to 0.12.
- Tree traversal and collapsed tree
- Picking the top k on each level from the top down, or sorting all nodes in one list and taking them up to a token limit. For example, for the all-deadlines question the collapsed tree takes 5 nodes within 100 tokens, A first.
- Tree, node, leaf, root
- A structure with levels: a node is a box, a leaf is a node at the bottom, and the root is the node at the top. For example, 2 summaries above 6 chunks (the leaves), and 1 root above them.
- UMAP
- Turning vectors into ones with fewer numbers while keeping nearby points nearby. For example, RAPTOR's code makes every vector 10 numbers long.
- Vector
- A list of numbers with a fixed length, such as [0.9, 0.1]. With 2 numbers, it is an arrow on a plane.
- Vocabulary
- The model's fixed list of tokens, each with an ID number. For example, it contains Wednesday but not Wednesdayy.
- Win rate
- The percentage of judgments that favour one of two systems, where 50% means a tie. For example, about 450 of 625 judgments, or 72%.
- Word overlap
- How many of the query's words appear in the passage. The simplest score. For example, the queue log has office and hour, so its score is 2.
- Word type and token in context
- A word as listed in the dictionary, and one use of it in a sentence. For example, late is a word type, and its use in
Late days may not be applied
is a token in context. - Zero-shot
- Doing a task from an instruction alone, without training on that task. For example, the prompt line “Answer the question using the passage below”.