TL;DR
We took one of our own production systems apart and costed it line by line. It was running at roughly $4,080 a month. It now runs at about $865, and none of the savings came from a cheaper provider.
The finding that reordered our own engineering priorities: 53% of the entire input bill was one 4,000-token prompt, re-sent on every step of every run. Not the retrieved documents, not the conversation, not the model choice. The same instructions, over and over, at full price.
Here is the system, the full LLM cost breakdown line by line, and what each of the five levers actually removed.
Want this done to your stack rather than read about it? We instrument cost per task before the model choice on every build. See how we approach AI development.
Key Takeaways
- 79% of the spend was input tokens, and only 16% of those tokens were genuinely new content.
- The single largest line was a static prompt re-sent every step. Caching it was the cheapest fix available and we had not done it.
- Fixing the loop before touching the model saved more than switching models would have.
- The levers multiply rather than add: 31%, then 40%, then 48% of what was left, which compounds to 79%.
- Lever five told us not to self-host, and that was the correct answer, not a failure of the exercise.
The system we costed
A support triage assistant. It reads an inbound ticket, searches our documentation, drafts a reply, and escalates when it is not confident.
| Before | |
|---|---|
| Runs per month | 30,000 tickets |
| Steps per run | 4 |
| System prompt and tool definitions | 4,000 tokens, re-sent every step |
| New content per step | 1,200 tokens |
| Output per step | 400 tokens |
| Context strategy | full transcript carried forward |
| Model | one strong model for all four steps |
| Runs needing a retry | 18% |
Nothing here is exotic. It is what you get when you build the obvious thing and ship it, which is exactly what we did.
The bill, line by line
Because the full transcript is carried, each step is more expensive than the one before it. That shape matters more than any single number.
| Step | Input tokens | Output tokens | Cost |
|---|---|---|---|
| 1 | 5,200 | 400 | $0.0216 |
| 2 | 6,800 | 400 | $0.0264 |
| 3 | 8,400 | 400 | $0.0312 |
| 4 | 10,000 | 400 | $0.0360 |
| Run | 30,400 | 1,600 | $0.1152 |
At 30,000 runs and an 18% retry rate, that is $4,078 a month.
The same work as a single call would be $0.0216, so the loop costs 5.3x one call. That is a modest multiple by agent standards, and only because this loop is four steps. The 25x figure applies to the twelve-step loops that are far more common than people expect.
Lever 1 cost nothing and changed everything
We did not change a line of code for a week. We just attributed the spend properly: cost tagged by feature, by step, and by what the tokens contained.
Three things fell out of that, and all three were surprises:
79% of the bill was input. We had spent engineering time shortening the assistant's replies. That was work on 21% of the problem.
53% of the input was one static prompt. Four thousand tokens of instructions and tool definitions, re-sent on all four steps of all 30,000 runs. We were paying full price for the same text roughly 120,000 times a month.
Only 16% of input tokens were new. Everything else was either the prompt or context the model had already been sent earlier in the same run.
That is the entire case for measuring before optimising. Every intuition we had about where the money was going was wrong, and we build these systems for a living.
Lever 2: fix the loop, not the model
The four steps were: classify the ticket, search the docs, draft a reply, then check the draft against a quality bar.
Steps three and four did not need to be separate. Asking for the draft and its self-assessment as one structured response collapses them into a single call with no loss, and it removes the most expensive step in the run, the one carrying the most accumulated context.
Three steps instead of four: $4,078 to $2,804, a 31% cut.
Worth sitting with: that is a larger saving than most model switches would have produced, and it came from deleting a call rather than from pricing.
Lever 3: route the steps that do not need the good model
Classifying a ticket and turning it into a search query are not hard problems. We had them on the strong model because every step was on the strong model, which is not a decision so much as an absence of one.
Steps one and two moved to a small model at roughly a third of the input price. Step three, the customer-facing reply, stayed where it was.
$2,804 to $1,671, a 40% cut on what was left.
We checked this the way small language models vs LLMs recommends rather than by feel: the small model's failure modes on those two steps are structural, so they are detectable, which is the actual precondition for routing safely.
Lever 4: stop paying for the same tokens
Now the finding from lever 1 gets its fix.
The 4,000-token prefix is identical on every call, which makes it exactly what prompt caching exists for. Cache reads bill at about a tenth of the input price.
One thing had to be checked first. Caching only helps if requests arrive faster than the cache expires. At 30,000 runs a month we get a request every 86 seconds on average, comfortably inside a five-minute window, so nearly every call is a read rather than a write. Had the traffic been a tenth of that, this lever would have done very little, which is the TTL trap that makes caching backfire for low-volume systems.
We also put the two routed steps on structured output. They return a category and a search query, so 400 tokens of prose was 400 tokens of politeness. That drops to 120, and because outputs get carried forward, it shrinks every subsequent step too.
$1,671 to $865, a 48% cut on what was left.
Lever 5 said no, and that was the answer
The last lever is where inference runs. After the first four, the system uses around 282 million tokens a month.
The break-even for self-hosting sits near 800 million tokens a month once you include GPU utilisation, redundancy and the engineering hours nobody puts in the spreadsheet. We are at roughly a third of it.
So we do not self-host, and we will not until volume roughly triples. A lever that returns "not yet" has still done its job. The failure mode is not checking, and then discovering two years later that you have been buying tokens at retail past the point where it made sense.
Note the ordering effect, too: we would have been further from break-even after optimising, not closer. Cutting the bill by 79% also cut the volume that self-hosting needs in order to pay off. Levers interact, and not always in the direction you want.
What is left, and what we would do next
The remaining $865 is not evenly spread.
Step three is now 77% of the run: the customer-facing reply, on the strong model, deliberately. The two routed steps together are less than a quarter.
That is the signal that this round is finished. The bill has concentrated into the step where spending money is the point, and any further cut would be a quality decision rather than an efficiency one. The next real saving is not a cost lever at all, it is raising the success rate so fewer runs happen twice.
What we would tell ourselves at the start
- Instrument before optimising. Every intuition we had was wrong, and a week of measurement was worth more than a month of guessing.
- Look at the loop before the model. Deleting a call beat every pricing decision available to us.
- Cache anything static and repeated, immediately. It was the largest single line and the cheapest thing to fix. We had simply never looked.
- Check the TTL against real request spacing. Caching is close to free money at our volume and close to worthless at a tenth of it.
- Run lever five even when you expect a no. The answer changes as you grow, and the cost of checking is an afternoon.
Run this on your own system
The arithmetic is not the hard part. Attribution is. If you can answer "which feature produced this invoice line" you can do everything above in a week.
Our calculators take the same inputs: the agent cost estimator for the loop, the caching calculator for the prefix, and the break-even model for lever five. The full method sits in LLM cost optimization.
Rather have us do it? We run this teardown as a fixed-scope engagement and instrument cost per task so the number stays visible afterwards. Tell us what your invoice looks like.
Frequently asked questions
How much can a typical AI system realistically be cut?
Between half and four fifths, in our experience, when nobody has looked at it before. The savings come from the same places nearly every time: a loop doing more calls than it needs, a static prompt paid for repeatedly, and every step running on the strongest model by default rather than by decision.
Should I switch to a cheaper provider first?
Almost never. Provider pricing is the flattest lever available and the most disruptive to change, since it means re-evaluating every prompt against a different model's behaviour. Fixing the loop cut more here than any switch would have, and it required no migration, no re-evaluation and no risk to output quality. Change providers when you have a reason beyond price.
Why is input more expensive than output when output bills more per token?
Because volume beats unit price. Output costs four to five times more per token, but this system sent nineteen input tokens for every output token. That ratio is typical for anything doing retrieval or running a loop, which is why token optimization so often targets the wrong side.
Does prompt caching always help?
No. It helps when a stable prefix is re-sent faster than the cache expires. At one request every 86 seconds against a five-minute window it is close to free money. At one request an hour, every call is a cache write and you pay a premium for nothing.
How long did all of this take?
About a week of measurement and two weeks of changes, spread around other work. The measurement week was the one that mattered: it decided the order of everything after it, and the order is what makes the savings multiply rather than overlap. It is also the week most teams skip, which is why they end up optimising whichever component they happened to think of first.
