LLM Cost Optimization: Your AI Bill Is the New Burn Rate

LLM cost optimization in practice. How model routing, prompt caching and waste removal cut a four-person team's $113K monthly AI bill, without losing quality.

TL;DR

LLM cost optimization is the practice of cutting what you spend on model inference without cutting output quality. Three levers do most of the work: route each request to the smallest model that can finish it, cache the fixed parts of your prompts, and delete the retries and oversized context windows nobody is measuring.

It matters because an LLM bill is not a subscription. It behaves like variable cost of goods sold, except the price is set by someone else and it scales with your success, so growth and burn now rise on the same curve. In April 2026 a four-person startup paid $113,421.87 for a single month of inference.

Underneath all three levers sits the thing that actually makes you defensible: owning the proprietary context that lets a small, specialised model beat a general-purpose frontier one on your task.

Building agents that have to survive their own invoice? Our AI agent development work starts from cost per completed task, not model benchmarks.

Key Takeaways

LLM costs scale with usage, so revenue growth and inference spend compound together.
One documented startup went from $27,690 to $113,421 in three months, roughly 4x.
Small models under 10B parameters run 10x to 30x cheaper than frontier siblings on narrow tasks.
Prompt caching bills repeated tokens at about 10% of standard input price, breaking even after roughly two reads.
A cheap model that fails and retries three times costs more than one strong call, so route on cost per completed task, not cost per token.
Cost control keeps you alive; a context moat is what keeps you fundable.

Three consecutive 2026 AI invoices for a four-person startup: $27,690 in February, $51,217 in March, $113,421 in April, a 4.1x rise in three months

In April 2026, a four-person startup got an invoice for $113,421.87. Not for salaries. Not for office space. For LLM inference. The company, Swan AI, builds sales agents, and its founder posted the receipt as a badge of honour, proof his tiny team was punching above its weight. Read the receipts in order and a different story shows up. February: $27,690. March: $51,217. April: $113,421. That is not a spend curve. That is a fuse.

You know this number, or a smaller cousin of it. The API line item that used to round to zero now has its own dashboard. Welcome to the discipline nobody put on the roadmap: watching your model spend the way you once watched your runway.

Why LLM costs behave differently from SaaS spend

For two years the story was that AI would replace headcount. The math has partly inverted. Costs to run generative systems are rising faster than the revenue they produce, and the people paying the bills have noticed. Erik Sherman's May 2026 report in Forbes lays out the scale: global tech capex hit roughly $740 billion this year, up about 69% over 2025. That money has to earn a return, and increasingly it is being pushed downstream into the price of a token.

The signals are no longer subtle. In the same reporting, Uber's CTO is described as having exhausted his entire 2026 AI budget by the start of the second quarter. Microsoft cut off internal Claude Code access to trim operating expense. NVIDIA's own VP of applied deep learning put it plainly: "Our costs for my team have been more for AI than human." When the company selling the shovels says the shovels cost more than the diggers, pay attention.

Here is the part that should reframe how you plan. Every new user, every retry, every "let me also summarize that" feature is a metered inference call. Growth that would thrill a normal founder can quietly bankrupt an AI-native one. Swan's revenue was climbing. Its inference costs were climbing faster. Guess which one wins that race if nobody intervenes.

The two lines start together. What separates them is not price per token, it is volume.

The shape is the problem, not the sticker price. You cannot negotiate your way out of a curve that steepens every time the product succeeds. You have to change what the product does per request. That is what LLM cost optimization actually is.

LLM cost optimization: the three levers that move the bill

The response emerging across serious teams has a name now, AI FinOps. Strip away the branding and it is one idea: stop paying frontier prices for work a smaller, cheaper model can do. Most of what your product does all day is not hard reasoning. It is classification, extraction, routing, templated summaries, quick rewrites. Narrow, repetitive, boring. And you have been running all of it on a model built to write a legal brief.

1. Route every request to the smallest model that can finish it

This is the biggest lever, and it is backed by real research, not vendor decks. NVIDIA's 2025 paper, "Small Language Models Are the Future of Agentic AI," found that serving a small language model like Llama 3.1B can be 10x to 30x cheaper than its 405B sibling, in compute, latency and energy, for the narrow tasks that make up most agentic workloads. Under about 10 billion parameters, small models are frequently good enough for tool calls, structured output and routing steps. Sometimes they beat the big model, because a fine-tuned specialist outperforms a generalist on its home turf.

The architecture that ties this together is small-model-by-default, frontier-on-escalation. Your agent runs the cheap model on every step. A lightweight router watches for trouble, a low-confidence answer, a schema violation, a tool call that will not parse, and only then escalates that single turn to a frontier model. Most requests never leave the cheap lane.

The widths are the argument. The expensive lane is meant to look barely used.

One implementation note that catches teams out. Cheaper models are cheaper per token, but a weak model that fails and retries three times can cost more than one strong call. Route on measured success rate and total cost per completed task, not on sticker price per million tokens. The goal is the cheapest system that finishes the job, not the cheapest model on the menu. It is the same failure mode that strands most agent projects in pilot: the demo runs on the expensive model and nobody costs the production version until the invoice lands.

2. Prompt caching: stop paying twice for the same tokens

If your prompts share a fixed preamble, a system prompt, a rulebook, a long document, a few-shot block, you are paying full input price to re-read the same tokens on every call. Prompt caching fixes that. As of 2026 every major provider supports it, and cached tokens run at roughly 10% of the standard input price. Anthropic, OpenAI and Google all land near a 90% discount on the repeated portion. There is a small write premium the first time, so break-even is around two reads. After that, every repeated call is nearly free on the cached span. Stack caching with a batch API for non-urgent work and the savings on repeated tokens climb past 95%.

Only the first bar is a real cost. The other four are the ones you are choosing to pay.

3. Cut the waste you are not measuring

Retries with no backoff. A 40,000-token context window feeding a question that needs 800 tokens of it. Two agents summarizing the same thread. A "chattier" default temperature that triples output length. None of these show up in a demo. All of them show up on the invoice. The discipline is boring and it works: log cost per request, alert on the outliers, delete the loops.

Small model vs frontier model: a working guide

Task	Recommended tier	Why
Classification, tagging, intent detection	Small model (under 10B)	Narrow and high-volume, cheap to fine-tune; frontier reasoning is wasted here
Data extraction into a fixed schema	Small model (under 10B)	Structured output is a specialist task; a tuned small model often beats a generalist
Routing and tool selection in an agent	Small model plus router	Runs on every step, so 10x to 30x cheaper compounds fast at agent scale
Templated summaries, quick rewrites	Small model (under 10B)	Bounded and repetitive; the quality gap is negligible on most product surfaces
Open-ended reasoning, multi-step planning	Frontier model	This is what you are actually paying frontier prices for
Long-context synthesis across many documents	Frontier model, with caching	A genuine capability need; cache the shared context to blunt the cost
Low-confidence or failed small-model output	Escalate to frontier	Pay the premium only for the fraction of turns that need it

The point is not to never use a frontier model. It is to stop using one for everything.

The context moat: why cheap and defensible are the same problem

Cost control keeps you alive. It does not make you valuable. And in 2026 investors are drawing that line sharply.

The 2026 founder's guide from We Are Presta puts it bluntly: "The age of the 'wrapper' is over." A thin layer over someone else's API has no moat, because the foundation model can absorb your feature in a release note. What is defensible is what they call the context moat: owning the historical data, the user preferences and the industry workflows. The guide argues that ownership makes a startup far harder to displace than one that merely has a better fine-tune.

Read that against the cost problem and the two halves of the strategy click together. Your proprietary context, the workflow data, the labelled edge cases, the domain records nobody else has, is exactly what lets you fine-tune a small model that beats a generalist frontier model on your specific task. The moat and the margin come from the same asset. The teams that survive 2026 are not the ones with the best prompt. They are the ones who turned their private data into a cheap, specialised system that a $740-billion incumbent cannot trivially copy.

That same guide reports a quiet metric shift: founders now track cost per inference alongside CAC. Unit economics have moved inside the product. A feature that loses money on every call is not a feature. It is a liability with a nice UI.

This is also the honest test for whether you should build the thing at all. If your differentiator is the workflow rather than the data, an off-the-shelf tool usually wins on cost, and the cheapest inference is the call you never make.

Your LLM cost optimization checklist

Instrument first. You cannot cut what you cannot see. Log cost per request and cost per completed task before you touch a single model choice.
Find your top three cost sinks and ask, honestly, whether each one needs a frontier model. Most will not.
Move one high-volume, narrow task to a small model. Classification or extraction is the safest first move. Measure the quality gap. It is usually smaller than your fear of it.
Turn on prompt caching for any workload with a shared preamble. This is nearly free money and it ships in an afternoon.
Add a router. Small model by default, escalate on failure. This is the change that compounds.
Name your context moat. Write down the proprietary data or workflow that a foundation model cannot replicate. If you cannot name it, that is the real problem, and it is bigger than your bill.

The $113,000 invoice is not a horror story about one reckless founder. It is a preview. The frontier-priced-everything era is closing because the people funding it ran the numbers. The teams that make it through are the ones who treat inference like the metered, variable cost it always was, and who own something upstream of the model that keeps them worth funding when the price of intelligence stops falling. Right-size the model. Cache the repeats. Guard your context. The bill is a design problem. Design for it.

Need the architecture, not just the advice? We build cost-aware LLM systems end to end. See how we approach AI development, or tell us what your invoice looks like.

Frequently asked questions

What is LLM cost optimization?

LLM cost optimization is the practice of reducing what you spend on model inference without reducing output quality. In practice it means three things: routing each request to the smallest model that can complete it, caching the fixed portions of prompts so repeated tokens are not billed at full price, and removing the retries, oversized context windows and duplicated calls that never appear in a demo but always appear on the invoice.

What are AI inference costs?

Inference cost is what you pay each time a model produces an output, as opposed to the one-off cost of training it. Providers bill per million tokens, split between input and output, and output tokens are typically several times more expensive than input. Because every user action triggers its own inference call, this cost scales directly with usage, which is why it behaves like cost of goods sold rather than a fixed software subscription.

What is AI FinOps?

AI FinOps is the practice of treating model spend as a managed, measured operating cost rather than a fixed subscription. It means instrumenting cost per request and per completed task, routing work to the cheapest model that can finish it, and removing the waste that never shows up until the bill arrives.

How much can prompt caching actually save?

Cached tokens bill at roughly 10% of the standard input price across the major providers, so the repeated span of a prompt costs about 90% less. There is a small write premium on the first call, which means you break even after roughly two reads. Layering a batch API on non-urgent work pushes the saving on that repeated span past 95%.

When should I use a small model instead of a frontier model?

Use a small language model, meaning under about 10 billion parameters, for narrow and repetitive work: classification, tagging, intent detection, extraction into a fixed schema, tool selection, and templated rewrites. Keep a frontier model for open-ended reasoning, multi-step planning and long-context synthesis. NVIDIA's research puts the cost gap at 10x to 30x on those narrow tasks.

Is a cheaper model always cheaper?

No. Cost per token is not cost per outcome. A weak model that fails and retries three times can cost more than a single strong call, and it burns latency and engineering attention on top. Measure success rate and total cost per completed task, then pick the cheapest system that reliably finishes the job.

How do I measure LLM cost per request?

Attach a cost field to every model call at the point of invocation, capturing input tokens, output tokens, cached tokens and the model used, then aggregate by feature and by completed task rather than by raw call. The number that matters is what one finished user outcome costs, because that is the figure that has to sit below what the user pays you.

What is a context moat?

A context moat is the proprietary data a competitor cannot copy: your historical records, labelled edge cases, user preferences and industry-specific workflows. It matters twice over. It is what a foundation model cannot absorb in a release note, and it is what lets you fine-tune a small, cheap model that outperforms a general-purpose frontier model on your specific task. The defensibility and the margin come from the same asset.