RAG Cost Optimization: Where Retrieval Spend Actually Goes

TL;DR

RAG cost is decided by two numbers multiplied together: how many chunks you retrieve, and how big each chunk is. Everything else is rounding. Retrieve twenty 512-token chunks and you have sent roughly 10,000 input tokens before the system prompt, on every single question, forever.

The part teams miss is that RAG has three separate cost centres, and only one of them scales with traffic. Embedding is mostly a one-off, vector storage is a fixed monthly floor that bills on a quiet month exactly as it does on a busy one, and generation is the per-query line that dominates almost every real bill.

The reflex fix for expensive retrieval, cutting top-k, is also usually the right one, and it tends to make answers better rather than worse. The relevant passage stops competing with fifteen near-misses.

Retrieval that answers well and bills quietly? We make chunking, top-k and caching decisions on measured evidence rather than framework defaults. See how we build AI systems that hold up on the invoice.

Key Takeaways

• Top-k multiplied by chunk size sets the input tokens on every query. Halving either roughly halves generation cost.
• Generation is typically the overwhelming majority of a RAG bill; embedding and storage are small but fixed.
• Cutting from twenty chunks to five commonly saves 60% or more, and frequently improves answer quality.
• Chunk overlap is charged twice over: it inflates embedding tokens and vector count together.
• Rerankers bill per search, not per document, and chunks over ~500 tokens are auto-split so each piece counts.
• Long context costs one to three orders of magnitude more per query than retrieval, until prompt caching narrows it.

The three cost centres of a RAG pipeline

The cost of a RAG pipeline is almost always quoted as one number. It is three, and they behave completely differently, which is why teams model one of them and get surprised by the other two.

The consequence worth internalising: a RAG system has a floor. If your traffic halves, your bill does not, because storage and any always-on infrastructure bill the same. That is a different shape from an agent, where cost is almost entirely variable, and it changes how you should think about a quiet month.

Why top-k times chunk size is the whole game

Here is the arithmetic that decides a RAG bill, and it fits on one line:

input tokens per query = system prompt + (top-k x chunk size) + question

That product is your RAG token usage, and it is the number worth putting on a dashboard.

With a 400-token system prompt, top-k of 20 and 512-token chunks, that is roughly 10,700 input tokens per question. The user typed maybe fifteen words. Everything else is retrieved context, and you pay for all of it whether the model uses it or not.

Two things follow, and the second one surprises people:

• The corpus size barely matters for per-query cost. Ten thousand documents or ten million, you still send top-k chunks. Corpus size drives embedding and storage, which are the small lines. This is exactly backwards from most people's intuition.
• Cutting top-k is nearly free quality-wise, and often positive. The relevant passage is usually in the top few results. Chunks six through twenty are mostly near-misses that dilute attention and give the model more opportunity to cite the wrong thing.

Put your own corpus and traffic in and the calculator will split the bill three ways and price the top-k change specifically.

Chunk size: the setting that moves everything

Chunk size looks like a retrieval-quality knob. It is also a cost knob, and it pulls in more than one direction at once.

Smaller chunks send fewer tokens per retrieved result, which lowers generation cost proportionally at a fixed top-k. But they produce more chunks overall, which raises vector count and storage, and they carry less context each, which can push you to raise top-k to compensate. Raise top-k by the same factor you cut chunk size and you have saved nothing.

Larger chunks send more tokens per result and cost more per query, but each result carries more context so a lower top-k often suffices.

The number to optimise is neither one alone. It is tokens sent per query, which is the product. Tune for the smallest product that still answers correctly, and check the answer quality rather than assuming.

One easily-missed detail: chunk overlap is charged twice. Overlapping text is embedded more than once and stored more than once, so a 20% overlap raises both embedding tokens and vector count by 20%. Those are the cheap lines, so it rarely matters much, but it is pure waste if the overlap is not earning better retrieval.

Reranking: priced per search, not per document

This one catches teams out because the pricing model does not work the way the mental model does.

Rerankers bill per search, where a search is one query plus up to roughly 100 documents. That makes reranking cheap in the normal case: a hundred thousand queries against a 50-candidate set is one search each, so a hundred searches per thousand queries.

Two things quietly multiply it:

• A candidate set above 100 documents costs more than one search. Fetching 150 candidates is two searches, not one and a half.
• Chunks longer than about 500 tokens are automatically split, and each piece counts as its own document. So raising chunk size from 400 to 600 tokens can double your rerank bill without changing a single other setting, because 50 candidates suddenly count as 100 documents.

The strategic point: a reranker only earns its cost if it lets you send fewer chunks to the model. Fetch fifty candidates, rerank, send the best four, and you have traded a small fixed cost for a large generation saving. Fetch fifty, rerank, and still send twenty, and you have added cost for nothing.

RAG vs long context

The RAG vs long context cost question is the one that keeps coming back, and it deserves a real answer rather than a slogan.

Now that context windows run to hundreds of thousands of tokens, the obvious question is whether retrieval is still worth the machinery. Why not send the whole corpus every time?

The measured answer is that retrieval is dramatically cheaper per query. Research comparing the two architectures on document-grounded generation found long-context prompting processing around 247,000 input tokens per query against roughly 8,500 for retrieval, with the cost difference landing at more than an order of magnitude and, in the extreme, around three. Latency follows the same shape: tens of seconds against roughly one.

But prompt caching changes the arithmetic, and this is the part most comparisons omit. If your corpus is stable, the stuffed context is a fixed prefix, which is exactly what caching is for. Cached tokens bill at roughly a tenth of standard input price, so long context stops being absurd and becomes merely expensive.

That makes it a real decision rather than an obvious one:

The honest summary: retrieval wins on cost at scale and wins on provenance always, while long context plus caching is genuinely competitive for a small corpus that does not change. Anyone telling you one of these always wins is selling something. The caching mechanics and break-even math are in our prompt caching guide.

When retrieval feeds an agent

One combination deserves a warning, because the two cost structures multiply rather than add.

If retrieval happens inside an agent loop, every step retrieves and every step re-sends everything the previous steps retrieved. A twelve-step agent doing retrieval at each step is not paying twelve times the retrieval cost. It is paying retrieval cost per step plus the accumulated weight of all previously retrieved chunks travelling again on every subsequent step.

Two rules keep it under control. Retrieve once per task where you can, not once per step. And drop retrieved chunks from the transcript once they have been used, rather than carrying them for the rest of the run. The compounding arithmetic behind that is in AI agent costs.

How we approach retrieval cost at HorizonLux

HorizonLux is a software development company specialising in AI, and RAG systems are a large share of what we build, so these are working defaults rather than recommendations we have read about.

• Top-k is tuned against measured answer quality, not set to the framework default and forgotten. We look for the point where quality starts to degrade, then step back one, and that number is almost always lower than teams expect.
• Chunking is costed before it ships. Tokens sent per query is a design output, not something discovered from the first invoice.
• Retrieval inside agent loops is treated as a red flag until someone can say why it must happen per step rather than once per task.
• The system prompt is ordered for caching from the first commit, stable content first, retrieved chunks last, so caching remains available even if we do not enable it on day one.
• The three cost centres are tracked separately. A rising bill that is storage means something completely different from a rising bill that is generation.

What to do this week

• Print your actual input tokens per query. System prompt plus top-k times chunk size. If it is over 10,000 for a question a human could answer from one paragraph, you have found your problem.
• Halve top-k on a copy and run your evaluation set. Compare answer quality honestly before assuming it got worse.
• Check whether your reranker changed your top-k. If it did not, it is cost without benefit.
• Check your chunk size against 500 tokens if you rerank, because crossing that line silently doubles the document count.
• Decide RAG versus long context on your actual corpus size and churn, not on which is fashionable.

Retrieval is one of the cheapest architectures available for grounding a model in your own data, and most systems still overpay for it by a factor of three or four. Almost all of that is one number set too high on the first day and never revisited. The wider sequence this fits into is our guide to LLM cost optimization, and if you are still working out which part of your stack is responsible, start with why is my AI bill so high.

Want retrieval that holds up on quality and on cost? We design the chunking, retrieval and caching decisions together, because changing one moves the others. See how we approach AI development, or tell us what your invoice looks like.

Frequently asked questions

What is RAG cost optimization?

RAG cost optimization is reducing what a retrieval pipeline spends without reducing answer quality. In practice it concentrates on one product: top-k multiplied by chunk size, which sets the input tokens on every query. Secondary levers are chunk overlap, which inflates embedding and storage, reranking configuration, and answer length. Corpus size matters far less than most teams assume, because it drives only the small fixed lines.

What drives the cost of a RAG pipeline?

Generation dominates almost every real bill, because each query sends top-k chunks through the model. Embedding is mostly a one-off at ingest plus a smaller ongoing cost as documents change, and vector storage is a fixed monthly amount that bills whether or not anyone asks a question. A rough rule: if your bill moves with traffic it is generation, and if it does not, it is storage.

How do I reduce RAG costs without hurting answers?

Cut top-k first. Most systems retrieve several times more than the answer needs, and reducing it usually improves answers because the relevant passage stops competing with near-misses. Then check tokens sent per query, cap answer length, and make sure any reranker is actually letting you send fewer chunks. Only after that is it worth changing embedding models or vector stores.

Is RAG cheaper than long context?

Per query, almost always, and often by one to three orders of magnitude, because retrieval sends a few thousand tokens where long context sends the whole corpus. Prompt caching narrows the gap substantially for a corpus that rarely changes, since the stuffed context becomes a cacheable prefix billed at roughly a tenth of input price. Retrieval still wins on provenance and on any corpus larger than a context window.

Does chunk size affect RAG cost?

Yes, in both directions. Smaller chunks send fewer tokens per retrieved result but produce more vectors and may force a higher top-k, cancelling the saving. Larger chunks cost more per result but often need fewer of them. Optimise the product of top-k and chunk size, which is tokens sent per query, rather than either number on its own.

How much does reranking cost in RAG?

Less than most people fear, provided it is configured carefully. Rerankers bill per search, where a search is one query plus up to about 100 documents, so a typical 50-candidate setup is one search per query. It inflates when the candidate set exceeds 100 documents, or when chunks longer than roughly 500 tokens are auto-split and each piece counts separately. Reranking only pays if it lets you send fewer chunks onward.

Is embedding a large corpus expensive?

Rarely. Embedding models are priced around a hundredth of generation models per million tokens, so embedding a ten-million-token corpus costs a few dollars once. What recurs is churn: the share of documents changing each month and needing re-embedding. Chunk overlap raises both the initial and ongoing cost, since overlapping text is embedded more than once.

Sources

• "The Token Tax of Epistemic Accuracy: Comparing RAG and Long-Context Architectures for Document-Grounded Generative AI Applications" (arXiv, 2026)
• "Small Language Models Are the Future of Agentic AI," NVIDIA Research (Belcak et al., 2025)