DocsGuides

Evaluating RAG

You cannot improve what you do not measure. RAG has two failure modes — bad retrieval and bad generation — and each needs its own metrics. Vibes-checking on a handful of demo queries is the most expensive way to ship a regression.

The two layers, separately

Retrieval

recall@k, MRR, nDCG, hit-rate

Generation

faithfulness, answer relevance

End-to-end

Exact match, LLM-as-judge

Tooling

RAGAS, TruLens, DeepEval, Promptfoo

If end-to-end accuracy regresses, you need to know where. Retrieval metrics tell you whether the right chunk made it into the context window; generation metrics tell you whether the model used it. Mixing them up sends you debugging the wrong layer for days.

Retrieval metrics

You need a labeled set: (question, list of relevant chunk IDs). Build it once by hand or have an LLM propose questions from your chunks (synthetic eval set — fast but biased toward what the embedder already handles).

python

def recall_at_k(retrieved_ids, relevant_ids, k):
    return len(set(retrieved_ids[:k]) & set(relevant_ids)) / len(relevant_ids)

def mrr(retrieved_ids, relevant_ids):
    for rank, doc_id in enumerate(retrieved_ids, start=1):
        if doc_id in relevant_ids:
            return 1.0 / rank
    return 0.0

def ndcg_at_k(retrieved_ids, relevant_ids, k):
    import math
    dcg = sum(
        1 / math.log2(i + 2) for i, d in enumerate(retrieved_ids[:k]) if d in relevant_ids
    )
    ideal = sum(1 / math.log2(i + 2) for i in range(min(len(relevant_ids), k)))
    return dcg / ideal if ideal else 0.0

Hit-rate is enough for most teams

For early-stage RAG, just track hit-rate@k = "did any relevant chunk make it into top-k?". It's coarse but moves with real quality and is trivial to label.

Generation metrics (RAGAS)

RAGAS popularized four LLM-as-judge metrics, each computed on a single sample of (question, retrieved context, generated answer, ground-truth answer):

Faithfulness

Are claims in the answer supported by the context?

Answer relevance

Does the answer address the question?

Context precision

Are retrieved chunks ranked by usefulness?

Context recall

Do retrieved chunks cover the ground truth?

python

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset

ds = Dataset.from_list([
    {
        "question": q,
        "answer": generated_answer,
        "contexts": retrieved_chunks,        # list of strings
        "ground_truth": gold_answer,
    }
    for q, generated_answer, retrieved_chunks, gold_answer in samples
])

result = evaluate(ds, metrics=[
    faithfulness, answer_relevancy,
    context_precision, context_recall,
])

Faithfulness, in plain English

Faithfulness asks: break the answer into atomic claims; for each claim, can it be verified from the retrieved context?. Score = supported_claims / total_claims. It catches the most common RAG failure — the model "filling in" with parametric knowledge when the context is incomplete. This metric is the one regressions tend to show up in first, because everything else (recall, BLEU) can look fine while the model quietly hallucinates a date.

Watch out for LLM-as-judge bias

The judge is also a model

LLM judges have known biases: they prefer longer answers, prefer answers from the same model family, and miss subtle factual errors that a human catches. Calibrate by sampling 50–100 examples, labeling them yourself, and measuring judge ↔ human agreement (Cohen's κ). Re-calibrate when you change judge models.

A practical eval harness

python

def evaluate_pipeline(pipeline, eval_set):
    rows = []
    for ex in eval_set:
        retrieved = pipeline.retrieve(ex["question"])
        answer = pipeline.generate(ex["question"], retrieved)
        rows.append({
            "q": ex["question"],
            "answer": answer,
            "retrieved_ids": [r.id for r in retrieved],
            "relevant_ids": ex["relevant_ids"],
            "ground_truth": ex["answer"],
        })
    return {
        "recall@5":     mean(recall_at_k(r["retrieved_ids"], r["relevant_ids"], 5) for r in rows),
        "mrr":          mean(mrr(r["retrieved_ids"], r["relevant_ids"]) for r in rows),
        "faithfulness": ragas_faithfulness(rows),
        "answer_rel":   ragas_answer_relevance(rows),
    }

Run this on every change to embedder, chunker, or prompt. Block merges on regression of any metric > 2 standard deviations.

Quick Quiz

Test yourself · 3 questions

Q1.

Why measure retrieval and generation metrics separately?

Q2.

What does RAGAS faithfulness measure?

Q3.

What's a key bias of LLM-as-judge evaluation?