Token Optimization: Most Teams Cut the Wrong Tokens

TL;DR

Token optimization is the practice of reducing how many tokens a request costs without reducing the quality of the answer. Most of it is done on the wrong tokens.

Almost every guide starts with the prompt: shorten the system message, trim the few-shot examples, compress the instructions. That advice is not wrong, it is just aimed at the cheapest and usually smallest part of the request. Two facts reorder the whole exercise:

• Output tokens cost four to five times what input tokens cost on every major provider. Cutting 100 tokens of answer is worth roughly 400 to 500 tokens of prompt.
• In most production systems the prompt is not the big number. Carried conversation, retrieved chunks and repeated preambles are, and none of them respond to prompt editing.

So the useful order is: measure where the tokens actually are, cut output first because it is mispriced in your favour, then fix whatever structural thing is inflating input, and only then start compressing prose.

Paying for tokens nobody reads? We instrument tokens per completed task before touching a prompt. See how we build AI systems that stay cheap in production.

Key Takeaways

• Output tokens bill at roughly 4x to 5x input, so answer length is the most mispriced lever in the whole exercise.
• Before compressing prose, find out whether the prompt is even a meaningful share of your request.
• Setting max_tokens deliberately and asking for structured output are the fastest genuine wins available.
• Prompt compression tools can reach high ratios, but they add a model call and are worth it only on large, stable, repeated prefixes.
• Caching beats compressing for anything repeated: a cached token costs about a tenth, with no quality risk at all.
• The number to track is tokens per completed task, because tokens per call falls when retries rise.

What is token optimization?

Token optimization is reducing the tokens a system spends per completed task while holding output quality steady. It covers four distinct levers that get lumped together and should not be:

Most articles cover the fourth and skip the first two. That ordering is backwards, and the reason is that the fourth is the only one you can do by editing a string.

Output tokens are the mispriced lever

Start here, because it is the highest return per unit of effort and almost nobody leads with it.

Every major provider charges substantially more for tokens the model produces than for tokens it reads. The typical spread is four to five times. Flagship tiers sit around $2 to $3 per million input tokens against $10 to $15 per million output.

Three changes act on it directly, and all three ship in an afternoon:

Set max_tokens deliberately. Most codebases leave it at a default that permits far more than the interface will ever display. The model will happily fill the space. A cap is not a quality decision, it is a bound on the worst case.

Ask for structured output when the consumer is code. If a parser reads the response, prose around the JSON is pure waste: generated, billed, then discarded. Structured output modes remove the preamble, the explanation and the closing summary in one change.

Stop asking for reasoning nobody reads. "Explain your thinking" doubles or triples an answer. If the explanation is never surfaced to a user or logged for review, you are paying premium rates to generate text straight into a bit bucket.

Output token reduction is the rare optimisation with a large win, a low quality risk and no architectural change.

Find out where your tokens actually are

Before compressing anything, get the breakdown for one real request. This takes an afternoon and it usually ends the debate.

Log, for a single representative request: system prompt tokens, few-shot tokens, retrieved or carried context tokens, the user's actual message, and output tokens. Then ask which segment you were about to spend a week compressing.

If carried conversation dominates, this is an architecture problem and the fix is in AI agent costs. If retrieved chunks dominate, lower top-k first and read RAG cost optimization. If the system prompt genuinely dominates, you are in the minority, and caching will beat compressing it.

That last point deserves emphasis because it is the most common wasted effort in this whole area: if a span repeats, cache it rather than compress it. A cached token bills at roughly a tenth of the input rate with zero quality risk, which beats almost any compression ratio you could achieve, and it takes less work. The mechanics are in prompt caching.

Context window optimization

The input side is mostly a discipline problem rather than a writing problem, and the useful moves are structural.

Cap what travels. A 40,000-token context feeding a question that needs 800 tokens of it is not thoroughness, it is waste with good intentions. Context window optimization means deciding what a step actually needs, not sending everything available because the window allows it.

Summarise instead of carrying. Conversation summarization replaces older turns with a running summary past a threshold. Chat history compression of this kind flattens the growth curve without the model losing the thread, and it is the single most effective context change in any long-running system.

Prune consumed content. Context pruning and context trimming both mean the same practical thing: once a tool result has been used, it does not need to travel for the rest of the run. Most frameworks keep it forever because that is the simplest default.

Reduce context length by relevance, not by truncation. Cutting the oldest N tokens is crude and drops things the model still needs. Deciding what is relevant is better, and it is why retrieval exists in the first place.

Long context cost is worth naming plainly here: large windows are a capability, not a budget. The fact that you can send 200,000 tokens does not mean a question needs them, and context window cost scales linearly with every one you send.

Prompt compression: when it is worth it

Now the technique most articles open with, placed where it belongs.

Prompt compression uses a smaller model to compress prompt text, stripping redundant tokens while preserving meaning. Tools in this family report high ratios, with LLMLingua-style approaches claiming up to 20x on suitable inputs. It is real, it works, and it has a cost profile people ignore.

Three things to weigh before adopting it:

• It adds a model call. Compressing costs inference too. On short prompts the compressor can cost more than it saves.
• The published ratios sit past the knee. Modest compression is nearly free. Aggressive compression removes information, and the damage is invisible until an evaluation set catches it.
• If the prompt repeats, caching is strictly better. No quality risk, no extra call, roughly 90% off. Compress only what is large, variable and not cacheable.

Where compression genuinely earns its place: long, unique, single-use documents that will not repeat and therefore cannot be cached. That is a real case. It is just narrower than the enthusiasm suggests.

What token optimization cannot fix

Being honest about the ceiling matters, because teams sometimes spend months here and move the bill very little.

Token optimization reduces the size of requests. It does not reduce how many requests you send. If an agent loops fifteen times when it needed four, or a retry storm is billing three attempts per operation, no amount of prompt editing touches that. Those are loop problems, and they belong to a different lever.

The ordering that works is in the LLM cost optimization guide: fix the number of calls first, then the size of each one. Token optimization is squarely the second half of that sentence, and it is far more effective once the first half is done.

Measure tokens per completed task, not per call

One measurement note that changes how you read your own results.

Tokens per call is the obvious metric and it is misleading, because it falls when things go wrong. Trim a prompt too far, the model fails more often, retries rise, and your average tokens per call improves while your bill grows. Token efficiency measured per call rewards exactly the failure mode you are trying to avoid.

Tokens per completed task is the honest number. It counts every attempt that went into one finished outcome, so a compression change that raised the failure rate shows up immediately rather than looking like a win.

How we approach token budgets at HorizonLux

HorizonLux is a software development company specialising in AI, and this is one of the areas where the difference between a working system and an expensive one is almost entirely habit.

• Every prompt gets a token budget before it gets written. How many tokens this call is allowed to cost is a design input, not something discovered from an invoice.
• max_tokens is always set explicitly. Never left at a default, because a default is a decision made by someone who has never seen your interface.
• Structured output wherever a parser is the consumer. If code reads it, the model does not write prose around it.
• Stable content goes first in the prompt, so caching stays available whether or not it is switched on yet. Retrofitting prompt order later means rewriting every prompt in the codebase.
• Compression is a last resort, applied only to large, unique, uncacheable inputs, and always with an evaluation set to find the knee.

What to do this week

• Print the token breakdown of one real request, split into system, few-shot, carried context, user message and output. Do this before choosing a lever.
• Set max_tokens everywhere it is currently unset. It is the single fastest change here.
• Switch to structured output anywhere a parser consumes the response.
• Cache before you compress. If a span repeats, caching wins on cost and cannot hurt quality.
• Track tokens per completed task, not per call, so a compression change that raises retries cannot masquerade as a saving.

Most LLM token waste is not clever inefficiency. It is a default nobody changed, a preamble nobody reads, and a transcript nobody needed to send. Fix the number of calls first, then make each one carry less, and measure it on completed tasks so the numbers cannot flatter you.

Want the token budget designed in rather than discovered later? We set max_tokens, prompt order and context policy before the first prompt ships. See how we approach AI development, or tell us what your invoice looks like.

Frequently asked questions

What is token optimization?

Token optimization is reducing the tokens a system spends per completed task without reducing answer quality. It covers four distinct levers: controlling output length and format, fixing structural input such as carried conversation or retrieved chunks, caching anything that repeats, and compressing the prompt wording itself. Most guides cover only the last one, which is usually the smallest and cheapest part of a request.

How do I reduce token usage in an LLM app?

Start with output, because it bills at four to five times input: set max_tokens deliberately and request structured output wherever a parser reads the response. Then look at structural input, which is usually the largest share: trim carried conversation, lower retrieval top-k, and drop tool results once consumed. Cache any repeated prefix. Compress the prompt wording last, since it is the smallest lever and the only one carrying real quality risk.

Is prompt compression worth it?

Sometimes, and less often than the headline ratios suggest. Compression adds its own model call, so on short prompts it can cost more than it saves, and the impressive published ratios sit past the point where quality starts to fall. It genuinely earns its place on long, unique, single-use documents that cannot be cached. If the content repeats at all, caching beats it: roughly 90% off with no quality risk and no extra call.

What is context window optimization?

Deciding what a request actually needs rather than sending everything the window permits. In practice that means summarising older conversation turns past a threshold, dropping tool results once they have been consumed, retrieving fewer and better chunks, and choosing content by relevance rather than truncating the oldest tokens. A large context window is a capability, not a budget, and context window cost rises with every token you choose to send.

Do output tokens really cost more than input tokens?

Yes, typically four to five times more across the major providers, with flagship tiers around $2 to $3 per million input against $10 to $15 per million output. That asymmetry means answer length is a pricing decision rather than a style one, and that removing 100 tokens from a response is worth roughly 400 to 500 removed from a prompt. It is the most commonly overlooked fact in token optimization.

How do I measure token efficiency properly?

Track tokens per completed task rather than per call. Tokens per call is misleading because it improves when quality degrades: a prompt trimmed too far causes more failures, more retries, and a lower average per call while the total bill rises. Counting every attempt that contributed to one finished outcome exposes that immediately, which is why it is the only metric that cannot flatter a bad change.

Will token optimization fix a high AI bill on its own?

Only partly. Token optimization reduces the size of each request but not the number of requests, so it cannot touch an agent looping more than it should or an unlogged retry storm. Fix call volume first, then request size. Applied in that order the savings compound; applied in reverse you are making the wrong number smaller.

Sources

• "Token optimization: The backbone of effective prompt engineering," IBM Developer
• "LLM Token Optimization: Cut Costs and Latency in 2026," Redis
• "Towards Optimizing the Costs of LLM Usage" (arXiv, 2024)