Evaluating RAG

A wrong answer from a RAG system has two possible causes, and they need different fixes. Either the right passage was never retrieved, or it was retrieved and the model failed to use it. A single end-to-end score cannot distinguish them, so it cannot tell you what to do next.

So evaluate in two stages. Score retrieval against your labelled passages, then score the answer only on the questions where retrieval succeeded. The second number is the one that tells you about the model, and it is usually much better than the end-to-end score suggests.

• Faithfulness: is every claim supported by a retrieved passage? A judge with the passages in front of it does this well.
• Answer relevance: does it address the question asked, rather than something adjacent that was retrieved?
• Attribution: does the cited passage actually contain the claim? Cheap to check and frequently wrong.
• Abstention correctness: does it decline exactly on the questions nothing answers?

Faithfulness is the one worth building first, because it catches the failure that matters: an answer that reads well, cites a real passage, and states something that passage does not say. That survives casual review in a way an obviously wrong answer does not.