TL;DR
Three variables decide this, not preference: how large the corpus is, how often it changes, and how much of it a single query actually needs. Long context wins on small, stable corpora hit by repetitive queries. Retrieval wins when the corpus is large, changes often, or is only sparsely relevant per query, which describes most production systems. Fine-tuning teaches behavior and format, not facts, and answers this question far less often than its invoice suggests. A fourth option costs nothing to try first: better prompting and a few examples solves more of this than the framing usually admits.
This post prices all three, with real numbers, not just the decision tree the context engineering hub sketches in miniature.
Choosing between RAG, fine-tuning, and a bigger context window for your next build? We price every branch before writing a line of code, because the wrong choice here is the most expensive mistake in the whole system.
Key Takeaways
- The decision runs on three variables: corpus size, how often it changes, and how much of it a typical query actually touches, not on which technique sounds most sophisticated.
- Fine-tuning changes behavior, tone, and output format. It does not reliably teach new facts, and using it as a knowledge base is the single most common misapplication of the technique.
- Long context is the right call more often than the "RAG for everything" default suggests, specifically for small, stable corpora with repetitive queries, where prompt caching makes the re-sent bulk cheap.
- Retrieval wins on cost and freshness the moment a corpus is large, changes without a retrain, or is only sparsely relevant to any single query, which covers most real production systems.
- Before reaching for any of the three, a better prompt with a few concrete examples costs nothing to test and resolves more failures than teams expect.
The fork, precisely
Every knowledge-heavy LLM application faces the same choice: put the documents in the window every time, retrieve only the relevant slice per query, or train the knowledge into the model's weights. Each answers a different question. Long context and retrieval both keep knowledge external and feed it in at inference time; fine-tuning bakes it into the model itself, which is why it behaves differently on the axis that actually matters, freshness. Update a document and a RAG system picks it up on the next query. Update a document and a fine-tuned model keeps answering from what it learned at training time until you retrain.
Before any of the three, there is a fourth option worth ruling out first: a better prompt. Few-shot examples and clearer instructions fix a surprising share of what looks like a knowledge problem but is actually a phrasing problem, and it costs nothing to test before reaching for infrastructure.
The three variables that actually decide it
Corpus size. A handful of documents fits comfortably in a long context window, especially with prompt caching absorbing the repeated cost. Once the corpus grows past what fits, or past what stays affordable to re-send every call, retrieval becomes the only option that scales.
Change rate. A corpus that updates daily needs an approach that reflects updates without a retrain. Retrieval indexes new documents immediately. Long context re-sends whatever is current at call time. Fine-tuning is frozen the moment training ends, which makes it a poor fit for anything that changes faster than your retraining cadence.
Query relevance. If most queries only need a small slice of a large corpus, retrieving that slice costs far less than sending the whole thing, and answers more precisely because the model is not searching for the needle itself. If nearly every query needs most of the corpus, long context's simplicity starts to win back the ground retrieval gave it.
What each option actually costs
This is the part the SERP for this term skips, and the reason to price it rather than just describe it. Take a knowledge base of roughly 2,000 documents, a question answered against it a thousand times a day.
Long context, re-sending relevant chunks of the corpus with every call, costs the most per query at scale, though prompt caching narrows the gap substantially for a stable, repeated corpus by billing the cached portion at a fraction of full input price. It stays the cheapest and simplest option only while the corpus is small enough that this bulk is manageable.
Retrieval costs an embedding call plus a small number of retrieved tokens per query, plus the fixed cost of an index that updates as documents change. At the scale in this example, retrieval's marginal cost per query is a small fraction of resending the full corpus, and it does not degrade as the corpus grows, only the index does.
Fine-tuning costs the most upfront, training compute charged per token processed during training, before a single production query runs, and that cost has to amortize over query volume to make sense. It does not solve the freshness problem at all; a document added the day after training finishes is invisible to the model until the next retrain, which is why fine-tuning answers "how should it behave" far better than it answers "what does it currently know."
RAG vs fine-tuning vs long context, side by side
| Long context | Retrieval (RAG) | Fine-tuning | |
|---|---|---|---|
| Best corpus size | Small, fits the window | Large, any size with indexing | N/A, teaches behavior not facts |
| Handles updates | Immediately, re-sent each call | Immediately, index updates | Only after retraining |
| Cost shape | Scales with corpus size per call | Small marginal cost per query | Large upfront, amortized over volume |
| What it actually changes | Nothing, just adds context | Nothing, just adds context | The model's weights and behavior |
| Wrong for | Large or frequently changing corpora | Nothing structural, mostly implementation quality | Teaching new facts or frequently changing knowledge |
Frequently asked questions
Should I use RAG or fine-tuning for my LLM application?
Use RAG when the knowledge is factual, changes over time, or is too large to fit in a context window; RAG updates immediately when documents change. Use fine-tuning when the goal is changing behavior, tone, or output format rather than teaching facts, since fine-tuning does not reliably learn new knowledge and is frozen until the next retrain.
Is long context a replacement for RAG?
Only for small, stable corpora hit by repetitive queries, where prompt caching makes the re-sent bulk affordable. Once a corpus grows large, changes often, or only a small slice is relevant per query, long context's cost and precision both lose to retrieval, which is why most production systems still land on RAG.
Can fine-tuning replace RAG for keeping an AI up to date?
No. Fine-tuning bakes knowledge into the model's weights at training time and does not update until you retrain, which makes it a poor fit for anything that changes faster than your retraining schedule. RAG retrieves from current documents on every query, so an update is reflected on the very next call.
What is the cheapest way to add knowledge to an LLM application?
Often the cheapest first step is neither RAG nor fine-tuning: a better prompt with a few concrete examples costs nothing to test and resolves a meaningful share of failures that look like knowledge gaps but are actually phrasing or instruction problems.
Can I combine RAG, fine-tuning, and long context in one system?
Yes, and production systems often do. A fine-tuned model for consistent tone and format, retrieval for current factual knowledge, and long context reserved for a small set of documents every query genuinely needs are not mutually exclusive; each addresses a different axis of the same problem.
