TL;DR
Agentic RAG adds one thing traditional RAG does not have: a decision loop that can evaluate whether what it retrieved is actually sufficient, and re-query, reformulate, or pull from a different source if it is not. Traditional RAG retrieves once and generates from whatever came back, correct or not. Agentic RAG treats retrieval as a step an agent can inspect, judge, and repeat, the same way it would decide whether a tool call's result answered the question or needs a follow-up.
This post covers the architecture, chunking strategy folded in as it belongs, the contextual retrieval technique that fixes a specific class of chunking failure, and an honest answer to when the loop is worth the extra latency and cost. Retrieval's economics are already priced elsewhere in this site; this post is about retrieval quality, the biggest lever in the context engineering series.
Building a RAG pipeline that keeps failing on queries a single retrieval pass cannot answer? We design AI agents with retrieval loops that know when to ask again, not just when to answer.
Key Takeaways
- Traditional RAG is one retrieval, one generation. Agentic RAG inserts a decision point: is what I retrieved sufficient, and if not, what should I search for instead.
- The loop matters most on multi-part questions a single query cannot resolve, and least on simple lookups where one retrieval already has the answer.
- Contextual retrieval, prepending chunk-specific context before indexing, fixes a failure mode no re-query loop can: a chunk that was never findable because it lost its context when it was split out.
- Agentic RAG does not replace good chunking. It makes the pipeline more forgiving of imperfect chunking by giving it a second, third, or fourth attempt.
- The extra loop costs latency and tokens on every query that needs it. It is not free, and it is not always necessary.
What actually makes RAG "agentic"?
Traditional retrieval-augmented generation is a fixed pipeline: embed the query, retrieve the top-k chunks, stuff them into the prompt, generate an answer. It runs the same way regardless of whether the first retrieval actually contained what the question needed. If the answer wasn't in the top five chunks, the model either says so or, worse, generates something plausible from what it got anyway.
Agentic RAG breaks that fixed shape. An agent, not a pipeline, decides what to retrieve, evaluates whether the result is sufficient, and can act on the answer being no: reformulate the query, retrieve from a different source, decompose a multi-part question into separate searches, or run another pass with the gaps identified. NVIDIA frames it as the difference between static retrieval and dynamic knowledge access, retrieval treated as a capability the agent reasons about rather than a step that always runs once and never checks its own work.
Chunking strategy, briefly, because it still matters
Agentic RAG makes a pipeline more forgiving of imperfect chunking, it does not make chunking irrelevant. Fixed-size chunking, splitting text every N tokens regardless of meaning, is cheap and breaks sentences and ideas across chunk boundaries at random. Semantic chunking, splitting at natural topic or section boundaries, costs more to compute and retrieves more coherently. Overlap between adjacent chunks, typically ten to twenty percent, catches material that would otherwise be split exactly at the boundary a query needs.
None of this disappears once retrieval becomes agentic. A badly chunked corpus still returns worse candidates on the first pass; the loop just gets more chances to recover from it, at the cost of the extra queries that recovery takes. Good chunking and an agentic loop are complementary, not substitutes for each other.
Contextual retrieval: the failure a loop cannot fix
Some retrieval failures are not about the query, they are about the chunk itself losing meaning when it was split from its source. A sentence reading "the company's revenue grew 3% over the previous quarter" means nothing on its own once separated from the document that named the company and the quarter. Standard chunking produces exactly this: technically accurate text, contextually orphaned.
Anthropic's contextual retrieval technique fixes this at index time rather than query time: before embedding or indexing each chunk, a short, chunk-specific explanatory context is generated and prepended to it, naming the document, the section, and what the chunk is actually about. The published results are substantial, contextual embeddings alone cut failed retrievals by roughly a third, and combining contextual embeddings with contextual keyword search and reranking cut them by around half. No amount of re-querying fixes a chunk that was never retrievable in the first place; this is a fix applied before the agentic loop ever runs.
When is the agentic loop actually worth it?
Not every query needs it. A lookup with one clear answer, "what is our refund policy," retrieves correctly on the first pass most of the time, and a decision loop around it adds latency without adding accuracy. The loop earns its cost on questions a single retrieval genuinely cannot resolve: multi-part questions that need information from more than one source, ambiguous queries where the first retrieval's relevance is genuinely uncertain, and follow-ups where the agent needs to decide whether its own earlier retrieval still applies.
The honest failure mode in the other direction is looping when nothing new is being found, an agent re-querying a corpus that simply does not contain the answer, burning tokens and latency on retries that were never going to succeed. A capped retry count and an explicit "not found in the available sources" exit path belong in the design from the start, not added after the first runaway loop.
Traditional RAG vs agentic RAG
| Traditional RAG | Agentic RAG | |
|---|---|---|
| Retrieval passes | One, always | One or more, decided at runtime |
| Evaluates its own result | No | Yes, before generating |
| Handles multi-part questions | Poorly, one query for everything | Can decompose into separate searches |
| Cost per query | Fixed and low | Variable, higher on queries that need retries |
| Best for | Simple lookups, single clear source | Multi-hop questions, ambiguous queries |
| What it does not fix | Orphaned chunks, missing context | Same, unless paired with contextual retrieval |
Frequently asked questions
What is agentic RAG?
Agentic RAG is retrieval-augmented generation with a decision loop added: instead of retrieving once and generating from whatever came back, an agent evaluates whether the retrieved material actually answers the question and can reformulate the query, search a different source, or decompose the question before generating a final answer.
How is agentic RAG different from traditional RAG?
Traditional RAG runs a fixed pipeline, embed, retrieve, generate, once per query, regardless of whether the retrieval actually succeeded. Agentic RAG inserts an evaluation step after retrieval that can trigger a reformulated query or an additional search, so the pipeline can recover from a first attempt that missed.
Does agentic RAG replace the need for good chunking?
No. Poor chunking still produces worse candidates on the first retrieval pass. Agentic RAG makes the pipeline more forgiving by giving it additional attempts, but it does not fix a chunk that lost its meaning when it was split from its source document. That specific failure needs contextual retrieval, not more retries.
What is contextual retrieval and how does it help?
Contextual retrieval prepends a short, chunk-specific explanation to each chunk before it is indexed, naming what document and section it came from, so the chunk stays findable once separated from its original context. Anthropic's published testing found it cut failed retrievals substantially, roughly a third with contextual embeddings alone.
When should I use agentic RAG instead of a simple retrieval pipeline?
When queries are multi-part, ambiguous, or regularly fail on a single retrieval pass. Simple, single-source lookups rarely need the loop and pay its latency cost for no accuracy gain. Cap the retry count and add an explicit "not found" exit path so the loop cannot retry indefinitely against a corpus that does not contain the answer.
