Small Model Break-even Calculator

Roughly one divided by how many times cheaper it is per call. A small model that is 20x cheaper breaks even near a 5% success rate, one that is 10x cheaper near 10%, and one that is only 4x cheaper near 25%. That is far lower than most teams assume, because a failed cheap attempt costs almost nothing next to the frontier call it was trying to avoid.

Only when the two tiers are close in price. Below roughly a 3x gap, the retries genuinely can outweigh the discount, and the warning is sound. At a real small-versus-frontier gap of 20x it is essentially never true on tokens alone. The reasons not to route are latency, complexity and undetected failure, not cost.

On your own evaluation set, not from a benchmark. Take a few hundred real inputs from the task you want to route, run them through the small model, and have a validator or a human decide whether each output was acceptable. Benchmarks measure general capability; what matters here is accuracy on one narrow task, which is usually much higher than the benchmark suggests.

Only failures you can detect automatically. A schema violation, a tool call that will not parse, a low confidence score or a failed validation are all detectable, so they can trigger escalation. A confidently wrong answer that passes validation is not, and that is the real risk in routing: undetected failure reaches the user, whereas detected failure only costs a retry.

Usually one or two. Failures are not independent: an input the small model got wrong once is usually a hard one, so a model that failed twice is unlikely to succeed on the third try, and further attempts mostly add latency. Two attempts then escalate is a reasonable default, and the calculator lets you price the difference.