Lesson 04 · Measuring · 13 min

Retrieval quality

A RAG system has two components that fail differently, and evaluating the answer alone cannot tell you which one failed. Measure retrieval separately, and most debugging becomes straightforward.

Recall@kOf the questions you have labelled, how often does the passage that answers them appear in the top k? This is the ceiling on everything downstream — an answer that was never retrieved cannot be generated.

PrecisionOf the passages retrieved, how many are relevant. Low precision does not make an answer impossible; it makes it more expensive and more likely to be diluted.

Recall is the one to fix first, and it is the one k moves. Raising k from 3 to 8 nearly always improves recall, because the correct passage was often ranked fourth. It costs tokens, and it feeds the model more irrelevant text, which is the trade.

Building the labelled set is the unglamorous part and it is not optional. Take twenty real questions, find the passage that actually answers each, and record it. That is a morning's work and it converts retrieval from an impression into a number.

In this lab the answering passage is recorded for each question, so the badge tells you honestly whether retrieval found it. That is recall@k on a dataset of five — enough to demonstrate, nowhere near enough to conclude, which is exactly the point of the score-confidence dive.

Watch what happens on the multi-passage question. The answer needs the damage rule and the furniture collection fee, which live in different sections. Retrieve one and the answer reads as complete while being half right — the most dangerous failure mode retrieval has.

Checkpoint

Record recall at k=3 and at k=8 across the five questions.