Evaluating RAG

A RAG answer can be wrong in two places: retrieval didn't find the right chunk, or the model didn't use it properly. Measure them separately, or you won't know which to fix.

For retrieval, build a set of real questions, each with the chunk or document that answers it. Recall@k is the share of questions whose right chunk is in the top k results. Mean reciprocal rank (MRR) also rewards finding it near the top: 1 for first place, 1/2 for second, 1/3 for third, 0 if missing, averaged over the questions.

# where the right chunk came in the results for five test questions (None = not in the top 5)
ranks = [1, 3, None, 2, 1]

recall_at_3 = sum(1 for r in ranks if r is not None and r <= 3) / len(ranks)
mrr = sum(1 / r for r in ranks if r is not None) / len(ranks)
print(recall_at_3)
print(round(mrr, 3))

For answers, check: is it faithful (every claim supported by the retrieved sources)? Is it correct (matches the known answer)? Does it answer the question that was asked? Does it abstain when the answer isn't in the documents? Code checks (citations exist, quotes match) catch a lot; a model grader with a clear rubric can judge faithfulness, checked against people's judgements.

Include questions whose answer isn't in the documents. A system that never says "I can't find that" will score well on the answerable ones and invent answers for the rest.

Change one thing at a time (chunk size, k, the reranker, the prompt) and rerun both sets. A change that lifts recall@10 but drops answer faithfulness, because the prompt is now stuffed with near-misses, isn't a win.