DevPath
Tracks

Evaluating RAG Systems

Measure retrieval with recall@k, MRR, and nDCG, check answers for faithfulness, build a golden set, and learn the common RAG failure modes. All metrics implemented from scratch.

advanced50 min#rag#evaluation#recall#mrr#ndcg#faithfulness#golden-set

Prerequisites: The RAG Pipeline End to End, RAG: Chunking and Retrieval

Why it matters

A RAG demo that answers three hand-picked questions proves nothing. Real users ask thousands of things you did not try, and every change you make (chunk size, embedding model, prompt, reranker) can improve some questions and silently break others. Without measurement you are guessing, and arguments about "which setup feels better" never end. Evaluation turns tuning into engineering: change one thing, rerun a fixed test set, compare numbers.

Intuition

Treat a RAG system as two students working in sequence: a researcher who fetches pages and a writer who composes the answer. If the final answer is wrong, you need to know who failed. Did the researcher bring back the wrong pages (retrieval failure)? Or did the researcher bring the right page and the writer ignore or distort it (generation failure)? Evaluate the two parts separately, then evaluate the whole.

How it works

1. Retrieval metrics. You need a golden set: questions, each labelled with which chunks (or documents) contain the answer. Then for each question compare the ranked list your retriever returns with the labels.

  • recall@k: the fraction of the relevant items that appear in the top k. If the answer is not retrieved, the model cannot use it, so recall@k (with k equal to what you pass to the prompt) is the most important retrieval number.
  • precision@k: the fraction of the top k that is relevant. Low precision means noisy prompts.
  • MRR (mean reciprocal rank): average of 1 / rank of the first relevant result. It rewards putting a correct hit at the top; a first hit at rank 1 scores 1, rank 2 scores 0.5, none scores 0.
  • nDCG@k (normalised discounted cumulative gain): handles graded relevance (for example 3 for a perfect passage, 1 for partly useful) and rewards ranking good items higher. DCG is the sum over ranks i of rel_i / log2(i + 1); divide by the DCG of the ideal ordering to get a value between 0 and 1.

Here is how recall@3 and reciprocal rank are read off a ranked list. These are Retriever A's results for the four golden questions in the code below (q2 has two relevant chunks, b and c).

rank 1rank 2rank 3reciprocal rankrecall@3
q1 (a)
a
x
y
1/1
1.00
q2 (b, c)
x
b
y
1/2
0.50
q3 (d)
x
y
d
1/3
1.00
q4 (e)
x
y
z
0
0.00
Green cells are relevant hits. Reciprocal rank looks only at the first hit (q3's first hit is at rank 3, so 1/3); recall@3 counts how many relevant chunks made the top 3 (q2 finds 1 of 2 = 0.50; q4 finds none). Averaging the last two columns gives MRR 0.458 and recall@3 0.625.

2. Generation metrics.

  • Faithfulness (groundedness): is every claim in the answer supported by the retrieved context? Check sentence by sentence. This catches hallucination even when retrieval was fine.
  • Answer relevance and correctness: does it address the question, and does it match a reference answer?
  • Citation accuracy: do cited sources exist and support the claim?
  • Abstention quality: does the system say "I don't know" on unanswerable questions, and avoid abstaining on answerable ones?

These are usually judged by humans on a sample, or by a second model acting as a judge (LLM-as-judge) with a clear rubric. Judges are convenient but imperfect: they can be biased, inconsistent, and swayed by length, so calibrate them against human labels before trusting them.

3. Building a golden set.

  • Start with real questions from users, logs, or support tickets. Invented questions are too easy and too similar to the documents.
  • For each question record the relevant chunk ids (several may qualify) and, for generation tests, a short reference answer or key facts.
  • Include unanswerable questions with the expected behaviour "abstain", plus tricky ones: exact identifiers, negations, multi-hop questions that need two chunks, questions with outdated documents.
  • Aim for a few dozen at first and grow it. Whenever production reveals a failure, add it as a regression case.
  • Keep it fixed between experiments, hold part of it out so you do not tune to the test, and version it with the code.

The golden set powers a simple loop you rerun after every change.

Golden setquestions + labels
Run retriever
Score recall@k, MRR, nDCG
Run generator
Judge faithfulness
Compare to last runfail on regression
Change one thing, rerun the same questions, compare the numbers. The cheap retrieval metrics run on every commit; the model-judged step can run on a sample or nightly.

4. Failure modes to look for.

  • Right document exists, never retrieved: chunking, embedding mismatch, or missing keyword search.
  • Right chunk retrieved at rank 15, prompt sees the top 5: needs reranking or a larger candidate pool.
  • Chunk retrieved, answer ignores it: the prompt, too much noise, or the model choice.
  • Answer states things not in the context: hallucination or a weak abstain instruction.
  • Answers with an old policy: stale index or no document versioning.
  • Fails on "what changed between X and Y": a multi-hop question that needs several retrievals.

Each failure mode belongs to one stage, which is why you measure the stages separately.

Ingest and chunkinganswer never indexed, or split in half
Retrievalright chunk missing or ranked below k: low recall@k, low MRR
Prompt assemblyright chunk present but buried in noise
Generationignores the context or invents claims: low faithfulness
Retrieval failures show up in recall@k and MRR; generation failures show up in faithfulness. High recall with low faithfulness points at the bottom layer, as in the quiz below.
OperationTimeNotes
recall@k, MRR, nDCG for Q queriesO(Q * k)Cheap; run on every change
Model-judged faithfulnessQ model calls (or more)Costly; run on a sample or nightly
Human reviewMinutes per itemMost trustworthy, least scalable
Golden set maintenanceOngoingAdd every production failure

Code: metrics and a faithfulness check

Below, two retrievers are represented by their fixed ranked outputs for four golden questions (chunk ids are letters), so you can verify the arithmetic by hand. The faithfulness function is a stand-in for an LLM judge: it says a sentence is supported when most of its longer words appear in the context. A real judge would be a model call that receives the context, the sentence, and a rubric, and returns supported or not; only the body of supported changes.

import math
import re
 
GOLDEN = {"q1": {"a"}, "q2": {"b", "c"}, "q3": {"d"}, "q4": {"e"}}
RETRIEVER_A = {"q1": ["a", "x", "y"], "q2": ["x", "b", "y"], "q3": ["x", "y", "d"], "q4": ["x", "y", "z"]}
RETRIEVER_B = {"q1": ["a", "x", "y"], "q2": ["b", "c", "x"], "q3": ["d", "x", "y"], "q4": ["x", "e", "y"]}
 
 
def recall_at_k(ranked, relevant, k):
    return len(set(ranked[:k]) & relevant) / len(relevant)
 
 
def reciprocal_rank(ranked, relevant):
    for i, item in enumerate(ranked, start=1):
        if item in relevant:
            return 1 / i
    return 0.0
 
 
def dcg(gains):
    return sum(g / math.log2(i + 1) for i, g in enumerate(gains, start=1))
 
 
def ndcg_at_k(ranked, gains, k):
    got = dcg([gains.get(item, 0) for item in ranked[:k]])
    ideal = dcg(sorted(gains.values(), reverse=True)[:k])
    return got / ideal if ideal else 0.0
 
 
def evaluate(name, results, k=3):
    n = len(GOLDEN)
    rec = sum(recall_at_k(results[q], rel, k) for q, rel in GOLDEN.items()) / n
    mrr = sum(reciprocal_rank(results[q], rel) for q, rel in GOLDEN.items()) / n
    ndcg = sum(ndcg_at_k(results[q], {i: 1 for i in rel}, k) for q, rel in GOLDEN.items()) / n
    print(f"{name} recall@{k}={rec:.3f} mrr={mrr:.3f} ndcg@{k}={ndcg:.3f}")
 
 
def supported(sentence, context):
    """Stand-in for an LLM judge: most longer words must appear in the context."""
    ctx = set(re.findall(r"[a-z0-9]+", context.lower()))
    content = [w for w in re.findall(r"[a-z0-9]+", sentence.lower()) if len(w) >= 4]
    return bool(content) and sum(w in ctx for w in content) / len(content) >= 0.6
 
 
def faithfulness(answer, context):
    sentences = re.split(r"(?<=\.)\s+", answer.strip())
    return sum(supported(s, context) for s in sentences) / len(sentences)
 
 
evaluate("A", RETRIEVER_A)
evaluate("B", RETRIEVER_B)
print(f"graded ndcg@3 good order={ndcg_at_k(['p', 'q', 'r'], {'p': 3, 'r': 2}, 3):.3f}"
      f" swapped={ndcg_at_k(['q', 'r', 'p'], {'p': 3, 'r': 2}, 3):.3f}")
 
context = "Refunds are issued within 5 business days to the original payment method."
good = "Refunds are issued within 5 business days. They are sent to the original payment method."
bad = "Refunds are issued within 5 business days. Customers also receive a free gift card."
print(f"faithfulness good={faithfulness(good, context):.2f} bad={faithfulness(bad, context):.2f}")

Both versions print:

A recall@3=0.625 mrr=0.458 ndcg@3=0.472
B recall@3=1.000 mrr=0.875 ndcg@3=0.908
graded ndcg@3 good order=0.939 swapped=0.648
faithfulness good=1.00 bad=0.50

Retriever B beats A on every metric, and the ranked lists in the code tell you why: A misses the answer entirely for q4 and places d at rank 3 for q3. The graded example shows nDCG punishing an ordering that puts the best passage last. In the faithfulness test, the unsupported gift-card sentence drops the score to 0.50, which is the signal that the answer contains a claim that the context does not support.

A recall@3
0.625
B recall@3
1
A mrr
0.458
B mrr
0.875
A ndcg@3
0.472
B ndcg@3
0.908
The printed metric values for Retriever A (first bar of each pair) and Retriever B (second bar). B wins on all three, and the gap is largest on rank-sensitive metrics (MRR, nDCG) because B puts hits at the top.

Common pitfalls

  • Only testing end-to-end: a bad answer does not tell you which stage failed. Measure retrieval and generation separately.
  • Metrics without labels you trust: if the relevance labels are sloppy, every number built on them is noise. Review labels, and allow several valid chunks per question.
  • Measuring document recall, then chunking differently: after changing chunking, chunk ids change. Label by stable source spans or re-label.
  • Over-trusting an LLM judge: check it against human judgements on a sample, watch for length bias, and use clear rubrics.
  • Ignoring unanswerable questions: a system that always answers scores well on answerable questions and is dangerous in production. Test abstention explicitly.
  • Tiny golden sets: with ten questions, one flip moves a metric by ten points. Report counts, and grow the set.
  • No regression tracking: keep a history of scores per commit so a drop is traceable to a change.

Practice

Compute precision@k and recall@k by hand easy

Hint

Precision divides by k, recall divides by the number of relevant items.

Show solution

Suppose the relevant set is {b, c} and the top 3 is [x, b, y]. One relevant item is in the top 3, so recall@3 is 1/2 and precision@3 is 1/3. If the top 3 were [b, c, x] both relevant items are found: recall 1, precision 2/3. Precision can never reach 1 when there are fewer than k relevant items.

Compute MRR for three queries easy

Hint

Take the reciprocal of the rank of the first relevant result for each query, then average.

Show solution

First relevant ranks are 1, 3, and not found. The reciprocal ranks are 1, 1/3, and 0, so MRR is (1 + 0.333 + 0) / 3, about 0.444. MRR ignores everything after the first hit, so use it when one good passage is enough.

Why does nDCG use a log discount? medium

Hint

Compare the weight of rank 1, rank 2, and rank 10.

Show solution

The weights 1 / log2(i + 1) are 1.0, 0.63, 0.5 for ranks 1, 2, 3 and about 0.29 at rank 10. The drop is steep at the top, where users and prompts look, and gentle further down. Dividing by the ideal DCG normalises scores so queries with different numbers of relevant items can be averaged.

Design an evaluation for a support-bot RAG hard

Hint

Cover retrieval, generation, abstention, and regression, and decide what runs when.

Show solution

Collect 100 real questions, label relevant chunks and key facts, and include 20 unanswerable ones. On every commit run retrieval metrics (recall@k, MRR, nDCG) and fail the build on a significant drop. Nightly, run a judged faithfulness and correctness pass on all questions, with a weekly human spot check of 20 judged items to monitor judge accuracy. Track abstention precision and recall separately. Add each production complaint as a new case. Report results with the question count so small changes are not overinterpreted.

Interview questions

  1. How do you evaluate a RAG system? Separate retrieval (recall@k, MRR, nDCG on a labelled golden set) from generation (faithfulness, correctness, citations, abstention), then run end-to-end checks.
  2. recall@k versus MRR? Recall asks whether the needed information appears in the top k at all; MRR asks how high the first hit is. Recall bounds what the generator can possibly use.
  3. What is faithfulness? The fraction of claims in the answer supported by the retrieved context; it detects hallucination independent of retrieval quality.
  4. What are risks of LLM-as-judge? Bias, inconsistency, sensitivity to length and phrasing; calibrate against human labels and use rubrics.
  5. How do you build a golden set? From real questions, with labelled relevant chunks, unanswerable cases, and regression additions, versioned and partly held out.

Interviewers look for: evaluating stages separately, knowing the metrics precisely, treating the golden set as a living asset, and scepticism toward automated judges.

Quick check

Your answers are often wrong. recall@5 is 0.95 but faithfulness is 0.5. Where should you look first?

Finished this lesson?

Further resources

Related lessons