Small Model Break-even Calculator
A small model is cheaper per call but wrong more often. How much more often can it be wrong before the retries eat the discount?
Share of attempts that produce an acceptable answer, measured on your own evaluation set rather than guessed.
Then hand the task to the frontier model.
Refine the inputsHide the detail
Rarely 100%. Its own retries are priced in too.
Above 5% the routed path is cheaper per completed task. The small tier is 22x cheaper per call, and the break-even lands near 1 divided by that.
Cost per completed task, as the small model gets better
The dashed line is frontier-only. Everywhere the bars sit below it, routing is cheaper. Note how early that happens.
Why the answer is lower than you expect
Three things the usual advice gets backwards.
Break-even is about 1/N
If the small tier is N times cheaper per call, it needs to succeed roughly one time in N to pay for its own retries. A 20x gap breaks even near 5%.
A failed cheap call is cheap
The intuition that retries are expensive comes from imagining them at frontier prices. At a 20x gap, five failed small attempts still cost a quarter of one frontier call.
Undetected failure is the real cost
Escalation only works on failures you can see: schema violations, parse errors, low confidence. A confidently wrong answer costs you a user, not a retry.
The full argument is in small language models vs LLMs, and the wider sequence is in LLM cost optimization.
Small model routing FAQ
What success rate does a small model need to be worth using?+
Roughly one divided by how many times cheaper it is per call. A small model that is 20x cheaper breaks even near a 5% success rate, one that is 10x cheaper near 10%, and one that is only 4x cheaper near 25%. That is far lower than most teams assume, because a failed cheap attempt costs almost nothing next to the frontier call it was trying to avoid.
Is it true that a cheap model that retries costs more than one strong call?+
Only when the two tiers are close in price. Below roughly a 3x gap, the retries genuinely can outweigh the discount, and the warning is sound. At a real small-versus-frontier gap of 20x it is essentially never true on tokens alone. The reasons not to route are latency, complexity and undetected failure, not cost.
How do I measure a small model success rate?+
On your own evaluation set, not from a benchmark. Take a few hundred real inputs from the task you want to route, run them through the small model, and have a validator or a human decide whether each output was acceptable. Benchmarks measure general capability; what matters here is accuracy on one narrow task, which is usually much higher than the benchmark suggests.
What counts as a failure for routing purposes?+
Only failures you can detect automatically. A schema violation, a tool call that will not parse, a low confidence score or a failed validation are all detectable, so they can trigger escalation. A confidently wrong answer that passes validation is not, and that is the real risk in routing: undetected failure reaches the user, whereas detected failure only costs a retry.
How many attempts should I allow before escalating?+
Usually one or two. Because success is roughly independent across attempts on the same input, a small model that failed twice is unlikely to succeed on the third try, so further attempts mostly add latency. Two attempts then escalate is a reasonable default, and the calculator lets you price the difference.
Routing that holds up on quality, not just on price
The calculator shows when routing pays. We build the validators and escalation logic that make it safe. Book a free cost review.