ColBERT is a retriever that keeps a separate vector for every token of a passage. For each token of the query it finds the best match in the passage, and it adds up the scores of these matches to get the passage's score. All the passages' token vectors are built before any question arrives, so ColBERT stays fast. Khattab and Zaharia built it in 2020.
The need for it showed up at the end of lesson 12: a rare code like error 4012 gets lost in a passage's single vector. For the query error 4012, the answer is in a long FAQ, but a one-vector bi-encoder ranks a short, wrong passage above it. The cross-encoder from lesson 14 fixes this mistake, but it takes 10.7 ms per passage.
The ladder for this lesson
The ladder has 4 steps. Under each step you can see what you will be able to do after it.
One vector buries 4012Compute an average vector by hand, and explain why the wrong passage came out on top.
What you need first: Lesson 3 (dot product, cosine), Lesson 7 (token vectors, mean pooling), Lesson 13 (MRR@k, Recall@k) and Lesson 14 (cross-encoder, reranking). If you already know a step, skip to the next one. A word with a dotted underline and ▾ opens its definition when you click it.
1One vector buries 4012
A bi-encoder squeezes a whole passage into one vector, and in a long passage one important token can get lost in that vector. Many bi-encoders build this vector by . In lesson 7 that meant adding up each number across all the token vectors and dividing by the number of tokens. For example, the average of [0.9, 0.1] and [0.7, 0.3] is [0.8, 0.2].
Now take the query error 4012, which has 2 tokens. Each token's vector has 3 numbers. The first says how much the token is about errors or uploads, the second how much it is about the code 4012, and the third how much it is about other topics. The query's tokens are error = [1, 0, 0] and 4012 = [0, 1, 0]. my vectors
The course's long FAQ covers grading, late days and Zoom, and one line gives the fix for error 4012. A short upload note only says that upload errors happen. A real FAQ would have hundreds of tokens, but here it has 6 to keep the arithmetic small.
First number: only error and 4012 have a first number other than 0. 0.9 + 0.1 = 1.0, and 1.0 ÷ 6 = 0.167.
Second number: only 4012 has one, 0.95, and 0.95 ÷ 6 = 0.158.
Third number: 1 from each of the four filler tokens, plus 0.1 from error, gives 4.1. Then 4.1 ÷ 6 = 0.683.
The third number is the largest, so the FAQ's vector points mostly towards other topics. Now take the from lesson 3, which is the dot product of two vectors divided by the product of their lengths. For example, the cosine of [1, 0] and [1, 1] is 1 ÷ (1 × 1.414) = 0.707. The query's average has a cosine of 0.32 with the FAQ's average, and 0.73 with the upload note's.
So the wrong upload note comes out on top, and the FAQ with the answer falls behind. The FAQ did contain 4012, but in the average it was only 1 token out of 6. Lesson 14 handled this situation with , bringing in BM25. ColBERT takes a different route: it never averages the tokens at all.
2MaxSim: the best match for each query token
ColBERT keeps a separate vector for every token of a passage, and a separate vector for every token of the query. These are the token vectors from lesson 7, where each token's vector is shaped by the words around it. The only difference is that ColBERT does not average them.
As with DPR in lesson 12, the document encoder builds these vectors and stores them in an , before any question arrives. When a question arrives, the query encoder builds vectors only for the query's tokens.
The score takes two steps. First, for each query token, take its with every passage token and keep the largest. Then add up these largest numbers over all the query's tokens. This score is called MaxSim.
Think first
In the average, grading, late, days and zoom buried 4012. Will these four tokens also pull down the FAQ's MaxSim score?
Show the answer
No. All four have a dot product of 0 with both query tokens, and MaxSim keeps only the largest dot product for each query token. The FAQ has a match larger than 0 for both, so these four tokens never enter the sum.
In the table below, each row is one FAQ token and each column is one query token. Each cell is their dot product, and the largest cell in each column is shaded.
FAQ token
Query's error [1, 0, 0]
Query's 4012 [0, 1, 0]
grading, late, days, zoomeach [0, 0, 1]
0
0
error[0.9, 0, 0.1]
0.9
0
4012[0.1, 0.95, 0]
0.1
0.95
Largest in the column
0.9
0.95
For example, the dot product of the query's error and the FAQ's 4012 is 1 × 0.1 + 0 × 0.95 + 0 × 0 = 0.1. The largest number in the error column is 0.9, and in the 4012 column it is 0.95. So the FAQ's MaxSim is 0.9 + 0.95 = 1.85.
The upload note's table works the same way:
Upload note token
Query's error [1, 0, 0]
Query's 4012 [0, 1, 0]
upload[0.8, 0.1, 0.1]
0.8
0.1
errors[0.95, 0.05, 0]
0.95
0.05
happen[0.5, 0, 0.5]
0.5
0
Largest in the column
0.95
0.1
The upload note's MaxSim is 0.95 + 0.1 = 1.05, so now the FAQ with the answer is on top: 1.85 against 1.05. The upload note has no token that matches 4012, so its best number in the 4012 column is only 0.1.
The formula holds the same two steps. Stanford's slide writes the passage as d, for document:
Sq,d = ∑i ∈ q maxjqi · dj
qi
The vector of query token i. For example, error = [1, 0, 0].
dj
The vector of passage token j. For example, the FAQ's 4012 = [0.1, 0.95, 0].
maxj
The largest dot product over all passage tokens j. For the FAQ, 0.9 in the error column.
∑i ∈ q
The max for every query token i, then their . ∈ means “belongs to”. For the FAQ, 0.9 + 0.95 = 1.85.
Here the query's tokens and the passage's tokens are encoded separately, and they meet only at the last step, the scoring. This is called late interaction, and ColBERT is the retriever built on it. Stanford's slide gives the same definition:
Stanford's slide: Late Interaction. The definition is at the top, and the MaxSim formula is in the middle. Below, (d) in the red frame is ColBERT, (a) is a bi-encoder like lesson 12's, and (c) is a cross-encoder like lesson 14's.Stanford's slide: ColBERT's architecture. The query encoder fQ builds vectors for the query's tokens, and the document encoder fD builds the passage's, inside the Offline Indexing box. Each MaxSim box is one query token's max, and ∑ adds them up.
Think first
Add a third token to the query: upload = [1, 0, 0]. What is the upload note's MaxSim now, and the FAQ's?
Show the answer
The new token's dot products with the upload note are 0.8, 0.95 and 0.5. The largest is 0.95, from errors, not from upload. The upload note's MaxSim is 1.05 + 0.95 = 2.0. On the FAQ the new token's max is 0.9, so the FAQ is at 1.85 + 0.9 = 2.75 and still ahead.
Run the MaxSim table yourself
Choose a passage and a query. Here each row is a query token, and each column is a passage token.
ColBERT has a limit too. Its passage vectors are built without seeing the query, while in a cross-encoder the query's tokens look at the passage's tokens through . So a difference that only shows when both are read together, like an old opinion against real data, is something a cross-encoder can catch better.
3The cost: 100 times the vectors, but far less work
The price of keeping a vector per token is paid in the index: the number of vectors grows with tokens, not with passages. Suppose there are 60,000 passages with about 100 tokens each. A bi-encoder stores 60,000 vectors, and ColBERT stores 60,000 × 100 = 6 million, which is 100 times as many. The gap in bytes is smaller, because ColBERT's vectors are also smaller. my example
What we get in return is measured in two ways. The first is latency: the time from sending a query to getting the answer, in ms. For example, the cross-encoder from lesson 14 takes 10,700 ms for 1,000 passages, so its latency is 10,700 ms.
The second is the computer's work, counted in FLOPs: how many multiplications and additions it took. Our vectors have 3 numbers, so one dot product has 3 multiplications and 3 additions, which is 6 FLOPs. The FAQ's MaxSim had 2 × 6 = 12 dot products, so it took 12 × 6 = 72 FLOPs.
Stanford's slide measures ColBERT on MS MARCO. As lesson 14 showed, it holds real Bing search questions and 8.8 million passages. In the rerank rows, the model re-sorts BM25's top 1,000. is from lesson 13: one over the rank of the first correct passage, so rank 2 gives 0.5. The table writes it multiplied by 100.
Method
MRR@10
Latency
FLOPs per query
BM25 (official)
16.7
–
–
BERT-base cross-encoder, rerank
34.7
10,700 ms
97T
ColBERT, rerank
34.9
61 ms
7B
ColBERT, end-to-end
36.0
458 ms
–
T means trillion, 1012, and B means billion, 109. So 97T ÷ 7B = 97,000 ÷ 7 = about 13,900. So when reranking, ColBERT does about 13,900 times less work than the cross-encoder, with almost the same MRR@10: 34.9 against 34.7. Its latency is 61 ms instead of 10,700 ms.
The gap exists because the cross-encoder runs the whole of BERT on every passage. ColBERT runs BERT once on the query, and the passages' vectors are already built, so all that is left per passage is MaxSim.
In the end-to-end row, ColBERT does not rely on anyone else's list. It searches all 8.8 million passages itself, in 458 ms, with an MRR@10 of 36.0. The big gain is in the shortlist. tells you what share of the correct passages reached the top 1,000, so 1 out of 2 gives 0.5. ColBERT's Recall@1000 is 96.8, against 85.7 for BM25 (the one from a toolkit called Anserini), which is 11.1 points more.
4Cascade: the cheap step first, the precise step last
You now have four retrievers: BM25, DPR, ColBERT and the cross-encoder. Stanford's slide puts all four in one table:
Stanford's slide: Fast vs Slow Retrievers. The columns show what is built in advance, the time per query, and where each retriever wins and breaks. You have already read ColBERT's row: a vector per token, 458 ms, and an index that grows with tokens.
Real systems put them in a line. The cheapest retriever pulls a long shortlist from the whole corpus, and each later step does more expensive, more precise work on fewer passages. This line is called a cascade. A cascade could look like this: my example
BM25 + DPRtop 1,000 out of tens of millions, with RRF
ColBERT reranktop 50 out of 1,000
Cross-encodertop 5 out of 50
LLMreads 5 passages and answers
Each step's cost depends on the size of its list. ColBERT reranks 1,000 passages in about 61 ms. The cross-encoder takes 50 × 10.7 = 535 ms for 50 passages, while all 1,000 would have taken it 10,700 ms.
Each step relies on the previous step's shortlist, so the first step's recall is the ceiling for the whole cascade. This is the from lesson 14. If the right passage is not in the first step's list of 1,000, no later step can bring it back. That is why the first step uses both BM25 and DPR, with merging their lists.
Think first
Your corpus is product manuals, and users often search by model number, like XR-2209. What happens if you drop BM25 from the cascade's first step?
Show the answer
Only DPR is left in the first step, and it misses rare literals. Passages with XR-2209 never make the shortlist, and no later reranker can bring them back.
What the dense version adds
The dense version contains this whole lesson, followed by 6 more steps that take you to the level of the Stanford slides:
Step 5: the full maths of MaxSim, and why taking an average instead of the max brings the wrong passage back.
Step 6: four ways to bring query and passage together, and the paper's experiments.
Step 7: inside ColBERT's encoders, and the training loss.
Step 8: the index in bytes, FLOPs counted out, and where the 61 ms go.
Step 9: the two stages of end-to-end search: candidate passages first, then full MaxSim on them.
Step 10: Stanford's four ColBERT slides, which you read yourself.
A one-vector bi-encoder can bury an important token of a long passage in the average. For error 4012, the FAQ's cosine was 0.32, and the wrong upload note's was 0.73.
ColBERT keeps a vector per token, and MaxSim adds up each query token's largest dot product. The FAQ got 1.85, and the upload note got 1.05.
When reranking, ColBERT is as precise as a cross-encoder, 34.9 against 34.7, with about 13,900 times fewer FLOPs. The price is 100 times the vectors. In a cascade the cheap step comes first, and the precise step comes last.
New terms in this lesson
Late interaction
The query's and the passage's token vectors are built separately, and meet only at scoring. For example, ColBERT builds passage vectors offline, and computes the dot products when the query arrives.
MaxSim
Each query token's largest dot product with the passage's tokens, then the sum of these. For example, 0.9 + 0.95 = 1.85 for the course FAQ.
ColBERT
A late-interaction retriever that keeps a vector per token and scores with MaxSim. Built by Khattab and Zaharia in 2020. For example, its Recall@1000 on MS MARCO is 96.8.
Latency
The time from sending a query to getting the answer, in ms. For example, the cross-encoder's latency is 10,700 ms for 1,000 passages, and ColBERT's reranking latency is 61 ms.
FLOPs
How many multiplications and additions it took, that is, the computer's work. For example, a dot product of 3 numbers takes 6 FLOPs, and one of 768 numbers takes 1,536.
Cascade
A line of retrievers in which each later step does more precise, more expensive work on fewer passages. For example, from tens of millions to 1,000, then 50, then 5.