DocsGuides

Agentic RAG

Classic RAG is one retrieval, one generation. Agentic RAG lets the model decide when to retrieve, what to retrieve, whether the result is good enough, and whether to try again — a control loop wrapped around the same primitives.

Why move past one-shot RAG

A single retrieval can't handle multi-hop questions ("Who succeeded the CEO of the company that bought Whole Foods?" needs two lookups), can't recover from a bad retrieval, and can't decide that the question doesn't need retrieval at all. Agentic RAG adds three missing capabilities: plan, judge, and retry.

Planner

Decompose into sub-questions

Router

Pick the right tool / index

Critic

Score the retrieved context

Loop

Retry / refine until satisfied

Minimal agentic loop

python

def agentic_answer(question, max_steps=4):
    notes = []                                 # accumulated context
    for step in range(max_steps):
        plan = llm.complete(PLAN_PROMPT.format(q=question, notes=notes))
        if plan["action"] == "ANSWER":
            return llm.complete(ANSWER_PROMPT.format(q=question, notes=notes))
        if plan["action"] == "SEARCH":
            hits = vector_db.search(embed(plan["query"]), k=5)
            judge = llm.complete(JUDGE_PROMPT.format(q=plan["query"], hits=hits))
            if judge["useful"]:
                notes.extend(hits)
            # else: loop, try a different sub-query
    return llm.complete(ANSWER_PROMPT.format(q=question, notes=notes))

Cost and latency

Every step is at least one LLM call plus a retrieval. Agentic loops are 3–10× slower and more expensive than vanilla RAG. Cap max_steps, cache retrievals, and fall back to single-shot for easy queries (route by question complexity).

CRAG — Corrective RAG

CRAG (Yan et al., 2024) adds a lightweight retrieval evaluator that tags each retrieval as correct, ambiguous, or incorrect:

Correct

Refine: strip irrelevant sentences, generate

Ambiguous

Combine local retrieval + web search

Incorrect

Discard, fall back to web search

Evaluator

Small fine-tuned classifier, not an LLM

The key insight: don't trust the vector store blindly. When the index doesn't actually contain the answer, returning the top-k anyway leads the LLM into a confident hallucination. A cheap classifier that rejects bad retrievals is one of the highest-leverage additions you can make.

Self-RAG — let the model decide when to retrieve

Self-RAG (Asai et al., 2023) trains the LLM to emit special reflection tokens: [Retrieve] when it wants context, [IsRel] to rate each retrieved passage, [IsSup] to mark whether its generation is supported by the evidence, and [IsUse] for overall usefulness. Retrieval happens on demand instead of at every turn, and the model can choose to ignore irrelevant context.

python

# Conceptual decoding loop (Self-RAG)
tokens = []
while not done:
    next_tok = model.generate(prompt + tokens)
    if next_tok == "[Retrieve]":
        hits = vector_db.search(embed(current_subquery(tokens)), k=3)
        rated = [model.rate(h, current_subquery(tokens)) for h in hits]
        tokens.extend(format(rated))
    else:
        tokens.append(next_tok)

Multi-hop with query decomposition

For genuinely multi-step questions, decompose first, retrieve per sub-question, then synthesize. This is what powers most "deep research" features in production.

python

DECOMPOSE = """Break this question into 2–4 atomic sub-questions whose
answers, combined, fully answer the original. Output JSON list.

Q: {q}"""

def multi_hop(q):
    subs = json.loads(llm.complete(DECOMPOSE.format(q=q)))
    evidence = {s: vector_db.search(embed(s), k=3) for s in subs}
    return llm.complete(SYNTHESIS_PROMPT.format(q=q, evidence=evidence))

Quick Quiz

Test yourself · 3 questions

Q1.

What's the main downside of an agentic RAG loop vs. single-shot RAG?

Q2.

What does CRAG's retrieval evaluator try to prevent?

Q3.

What's distinctive about Self-RAG?