Token Optimization: Most Teams Cut the Wrong Tokens

TL;DR

Token optimization is the practice of reducing how many tokens a request costs without reducing the quality of the answer. Most of it is done on the wrong tokens.

Almost every guide starts with the prompt: shorten the system message, trim the few-shot examples, compress the instructions. That advice is not wrong, it is just aimed at the cheapest and usually smallest part of the request. Two facts reorder the whole exercise:

• Output tokens cost four to five times what input tokens cost on every major provider. Cutting 100 tokens of answer is worth roughly 400 to 500 tokens of prompt.
• In most production systems the prompt is not the big number. Carried conversation, retrieved chunks and repeated preambles are, and none of them respond to prompt editing.

So the useful order is: measure where the tokens actually are, cut output first because it is mispriced in your favour, then fix whatever structural thing is inflating input, and only then start compressing prose.

Paying for tokens nobody reads? We instrument tokens per completed task before touching a prompt. See how we build AI systems that stay cheap in production.

Key Takeaways

• Output tokens bill at roughly 4x to 5x input, so answer length is the most mispriced lever in the whole exercise.
• Before compressing prose, find out whether the prompt is even a meaningful share of your request.
• Setting max_tokens deliberately and asking for structured output are the fastest genuine wins available.
• Prompt compression tools can reach high ratios, but they add a model call and are worth it only on large, stable, repeated prefixes.
• Caching beats compressing for anything repeated: a cached token costs about a tenth, with no quality risk at all.
• The number to track is tokens per completed task, because tokens per call falls when retries rise.

What is token optimization?

Token optimization is reducing the tokens a system spends per completed task while holding output quality steady. It covers four distinct levers that get lumped together and should not be:

Most articles cover the fourth and skip the first two. That ordering is backwards, and the reason is that the fourth is the only one you can do by editing a string.

Output tokens are the mispriced lever

Start here, because it is the highest return per unit of effort and almost nobody leads with it.

Every major provider charges substantially more for tokens the model produces than for tokens it reads. The typical spread is four to five times. Flagship tiers sit around $2 to $3 per million input tokens against $10 to $15 per million output.

Three changes act on it directly, and all three ship in an afternoon:

Set max_tokens deliberately. Most codebases leave it at a default that permits far more than the interface will ever display. The model will happily fill the space. A cap is not a quality decision, it is a bound on the worst case.

Ask for structured output when the consumer is code. If a parser reads the response, prose around the JSON is pure waste: generated, billed, then discarded. Structured output modes remove the preamble, the explanation and the closing summary in one change.

Stop asking for reasoning nobody reads. "Explain your thinking" doubles or triples an answer. If the explanation is never surfaced to a user or logged for review, you are paying premium rates to generate text straight into a bit bucket.

Output token reduction is the rare optimisation with a large win, a low quality risk and no architectural change.

The token type you cannot see: reasoning tokens

There is now a third token category, and it is the fastest-growing line on 2026 invoices while being the only one you never read.

Reasoning models think before they answer. That internal chain, the plans considered and discarded, the checks run against their own logic, is generated token by token like any output, and it bills as output, at output rates, even though none of it appears in the response. On premium reasoning tiers the output multiple runs as high as 8x the input price, so the invisible tokens are also the most expensively priced ones you buy.

The scale is not marginal. Reasoning and agentic workloads consume 5x to 30x more tokens per task than the equivalent plain completion, and the FinOps Foundation identifies exactly this shift as the main reason 2024-era cost forecasts stopped working: the visible per-call price looks ordinary while the consumption profile underneath is a different animal.

Two controls exist, and most teams have touched neither:

• The effort or thinking-budget setting. Providers expose a dial for how much internal reasoning a request may spend. It defaults generously. A classification task on a reasoning model at default effort is paying for deliberation it does not need.
• Model class per task, again. The cheapest way to not pay for reasoning tokens is to not send routine work to a reasoning model. This is the routing argument with a sharper edge, because the price gap per completed task is wider than the sticker suggests.

If your bill rose after adopting a reasoning model and your dashboard shows modest response lengths, this is where the money went. The response was short. The thinking was not.

Find out where your tokens actually are

On our own bill, the largest single line was a prompt we re-sent on every turn. Nobody had looked, because nothing in the invoice pointed at it.

Before compressing anything, get the breakdown for one real request. This takes an afternoon and it usually ends the debate.

Log, for a single representative request: system prompt tokens, few-shot tokens, retrieved or carried context tokens, the user's actual message, and output tokens. Then ask which segment you were about to spend a week compressing.

If carried conversation dominates, this is an architecture problem and the fix is in AI agent costs. If retrieved chunks dominate, lower top-k first and read RAG cost optimization. If the system prompt genuinely dominates, you are in the minority, and caching will beat compressing it.

That last point deserves emphasis because it is the most common wasted effort in this whole area: if a span repeats, cache it rather than compress it. A cached token bills at roughly a tenth of the input rate with zero quality risk, which beats almost any compression ratio you could achieve, and it takes less work. The mechanics are in prompt caching.

Context window optimization

The input side is mostly a discipline problem rather than a writing problem, and the useful moves are structural.

Cap what travels. A 40,000-token context feeding a question that needs 800 tokens of it is not thoroughness, it is waste with good intentions. Context window optimization means deciding what a step actually needs, not sending everything available because the window allows it.

Summarise instead of carrying. Conversation summarization replaces older turns with a running summary past a threshold. Chat history compression of this kind flattens the growth curve without the model losing the thread, and it is the single most effective context change in any long-running system.

Prune consumed content. Context pruning and context trimming both mean the same practical thing: once a tool result has been used, it does not need to travel for the rest of the run. Most frameworks keep it forever because that is the simplest default.

Reduce context length by relevance, not by truncation. Cutting the oldest N tokens is crude and drops things the model still needs. Deciding what is relevant is better, and it is why retrieval exists in the first place.

Long context cost is worth naming plainly here: large windows are a capability, not a budget. The fact that you can send 200,000 tokens does not mean a question needs them, and context window cost scales linearly with every one you send.

Prompt compression: when it is worth it

Now the technique most articles open with, placed where it belongs.

Prompt compression uses a smaller model to compress prompt text, stripping redundant tokens while preserving meaning. Tools in this family report high ratios, with LLMLingua-style approaches claiming up to 20x on suitable inputs. It is real, it works, and it has a cost profile people ignore.

Three things to weigh before adopting it:

• It adds a model call. Compressing costs inference too. On short prompts the compressor can cost more than it saves.
• The published ratios sit past the knee. Modest compression is nearly free. Aggressive compression removes information, and the damage is invisible until an evaluation set catches it.
• If the prompt repeats, caching is strictly better. No quality risk, no extra call, roughly 90% off. Compress only what is large, variable and not cacheable.

Where compression genuinely earns its place: long, unique, single-use documents that will not repeat and therefore cannot be cached. That is a real case. It is just narrower than the enthusiasm suggests.

The token sink nobody audits: your tool definitions

Every optimization list covers prompts, context and output. Almost none covers the one that grows fastest in an agent, because it does not look like a prompt at all.

When a model chooses a tool, it reads every tool you offered it. The full schema of every function, including the ones it will never call on this request, is serialised into the prompt and billed as input tokens. Ten tools with detailed parameter descriptions is easily two to three thousand tokens riding along on every single call in the loop, whether one tool is used or none.

This compounds badly with the loop arithmetic. A twelve-step agent carrying 2,500 tokens of unused tool schemas pays for them twelve times per run, and pays again on every retry.

The scale of this is now measured rather than argued. Cloudflare reported an 81% token reduction on a multi-step task purely by changing how tools are exposed to the model. On an MCP server carrying 2,500 API endpoints, the flat tool definitions came to roughly 1.17 million tokens, compressed to about 1,000. Anthropic reported 98.7% on a Drive-to-Salesforce workflow under the same pattern.

Those are not typos. Tool definitions are the largest unexamined line item in most agent prompts.

Three mitigations, in order of effort, and then the change that removes the problem instead of shrinking it:

• Filter the tool list before the call. Pass only the tools plausibly relevant to the current step. Most agents can decide that cheaply from the task type, and it is the single largest cut available in a tool-heavy system.
• Make the system prompt conditional too. Instructions that read "when using the email tool, always…" are dead weight on every request where the email tool was not passed. Include those sections only when the matching tool is in the filtered list.
• Return structured errors, not bare failures. A 400 Bad Request tells the model nothing, so it guesses and retries, and each retry is a fresh full-price request. An error that states what was wrong and what a valid call looks like usually converts a retry loop into one corrected call. Error message design is token optimization, which is not where anyone thinks to look.

Note the pattern: none of these makes the prompt shorter. They make the number of tokens you send per completed task smaller, which is the thing that actually bills.

The format you serialise in is a cost decision

One more sink that hides in plain sight, because it looks like an engineering convention rather than a spend decision.

JSON is expensive to tokenise. Its punctuation and repeated key names cost tokens on every row, and for tabular data that overhead is most of the payload. CSV, TSV and formats designed for model consumption carry the same information in roughly 30% to 60% fewer tokens. If you are passing rows of anything, retrieved records, search results, tool output, the format you chose casually is now a recurring bill.

The same holds on the way out. Microsoft research found that function-calling-based structured output is materially more token-efficient than asking a model to generate free-form JSON, which is a second reason to prefer it beyond the reliability argument this post already made.

Two rules that cover most cases:

• Tabular data in, use a tabular format. Reserve JSON for genuinely nested structures where its shape earns the overhead.
• Structured data out, use the structured output feature rather than describing the schema in the prompt and hoping.

Neither changes what the model can do. Both change what you pay to say it.

What token optimization cannot fix

Being honest about the ceiling matters, because teams sometimes spend months here and move the bill very little.

Token optimization reduces the size of requests. It does not reduce how many requests you send. If an agent loops fifteen times when it needed four, or a retry storm is billing three attempts per operation, no amount of prompt editing touches that. Those are loop problems, and they belong to a different lever.

The ordering that works is in the LLM cost optimization guide: fix the number of calls first, then the size of each one. Token optimization is squarely the second half of that sentence, and it is far more effective once the first half is done.

The fix that removes the problem: let the model write code

Filtering makes the list shorter. There is a change that stops the list being sent at all, and Anthropic and Cloudflare arrived at it independently.

The standard pattern enumerates every tool as a schema and asks the model to pick one. The alternative gives it a small number of meta-tools and has it write code that calls your API, rather than selecting from a menu. The insight underneath is that models are stronger at writing code that calls tools than at choosing between tool definitions in function-calling syntax, and code the model writes does not require every endpoint to have been described in advance.

The cost consequence is the numbers above. Instead of paying for N schemas on every step of every run, you pay for two meta-tool definitions plus whatever code the model generates for that step. The saving does not scale with your effort; it scales with how many tools you had.

Three honest caveats, because this is an architecture change rather than a config flag:

• You need somewhere to run the code. A sandbox with the right network access and the right blast radius, which is real work and real risk surface.
• The model still needs to know your API's shape, through documentation retrieval or a discovery call. That cost does not vanish, it moves.
• It is not worth it at five tools. The pattern pays off as tool count grows, and the cliff is steep: the more endpoints you have, the worse enumeration gets and the better this looks.

If you have a handful of tools, filter the list. If you have dozens or hundreds, the enumeration pattern is the thing costing you money, not the prompt.

If your tokens are going into a coding agent

A growing share of people asking about token usage are not running an API integration at all. They are watching Claude Code, Copilot or Cursor burn through a budget, and the advice above applies with the emphasis moved.

The mechanics are the same, but three things dominate:

• The context is the codebase, and it accumulates. Every file read, every diff, every tool result stays in the transcript and is re-sent on the next turn. This is the agent loop compounding, not prompt bloat, and trimming your instructions will not touch it. Start fresh sessions at task boundaries rather than carrying one enormous thread.
• Caching is doing most of the work already. Coding agents keep a large, stable prefix and hit it constantly, which is close to the ideal caching shape. Check that your prefix really is stable: anything injected per turn near the top of the prompt quietly breaks the cache and you pay full price without noticing.
• Model choice per task, not per session. Asking for a rename or a formatting pass from the most capable model is the same mistake as routing everything to a frontier model in production, and the same arithmetic applies.

The honest framing: an agent that reads more files than it needs is not a prompt problem, it is a scope problem, and the fix is a tighter task rather than a shorter instruction.

Measure tokens per completed task, not per call

One measurement note that changes how you read your own results.

Tokens per call is the obvious metric and it is misleading, because it falls when things go wrong. Trim a prompt too far, the model fails more often, retries rise, and your average tokens per call improves while your bill grows. Token efficiency measured per call rewards exactly the failure mode you are trying to avoid. The full method, the five fields to log and what counts as completed, is in cost per task.

Tokens per completed task is the honest number. It counts every attempt that went into one finished outcome, so a compression change that raised the failure rate shows up immediately rather than looking like a win.

How we approach token budgets at HorizonLux

HorizonLux is a software development company specialising in AI, and this is one of the areas where the difference between a working system and an expensive one is almost entirely habit.

• Every prompt gets a token budget before it gets written. How many tokens this call is allowed to cost is a design input, not something discovered from an invoice.
• max_tokens is always set explicitly. Never left at a default, because a default is a decision made by someone who has never seen your interface.
• Structured output wherever a parser is the consumer. If code reads it, the model does not write prose around it.
• Stable content goes first in the prompt, so caching stays available whether or not it is switched on yet. Retrofitting prompt order later means rewriting every prompt in the codebase.
• Compression is a last resort, applied only to large, unique, uncacheable inputs, and always with an evaluation set to find the knee.

What to do this week

• Print the token breakdown of one real request, split into system, few-shot, carried context, user message and output. Do this before choosing a lever.
• Set max_tokens everywhere it is currently unset. It is the single fastest change here.
• Switch to structured output anywhere a parser consumes the response.
• Cache before you compress. If a span repeats, caching wins on cost and cannot hurt quality.
• Track tokens per completed task, not per call, so a compression change that raises retries cannot masquerade as a saving.

Most LLM token waste is not clever inefficiency. It is a default nobody changed, a preamble nobody reads, and a transcript nobody needed to send. Fix the number of calls first, then make each one carry less, and measure it on completed tasks so the numbers cannot flatter you.

Want the token budget designed in rather than discovered later? We set max_tokens, prompt order and context policy before the first prompt ships. See how we approach AI development, or tell us what your invoice looks like.

Frequently asked questions

What is token optimization?

Token optimization is reducing the tokens a system spends per completed task without reducing answer quality. It covers four distinct levers: controlling output length and format, fixing structural input such as carried conversation or retrieved chunks, caching anything that repeats, and compressing the prompt wording itself. Most guides cover only the last one, which is usually the smallest and cheapest part of a request.

How do I reduce token usage in an LLM app?

Start with output, because it bills at four to five times input: set max_tokens deliberately and request structured output wherever a parser reads the response. Then look at structural input, which is usually the largest share: trim carried conversation, lower retrieval top-k, and drop tool results once consumed. Cache any repeated prefix. Compress the prompt wording last, since it is the smallest lever and the only one carrying real quality risk.

Is prompt compression worth it?

Sometimes, and less often than the headline ratios suggest. Compression adds its own model call, so on short prompts it can cost more than it saves, and the impressive published ratios sit past the point where quality starts to fall. It genuinely earns its place on long, unique, single-use documents that cannot be cached. If the content repeats at all, caching beats it: roughly 90% off with no quality risk and no extra call.

What is context window optimization?

Deciding what a request actually needs rather than sending everything the window permits. In practice that means summarising older conversation turns past a threshold, dropping tool results once they have been consumed, retrieving fewer and better chunks, and choosing content by relevance rather than truncating the oldest tokens. A large context window is a capability, not a budget, and context window cost rises with every token you choose to send.

Do output tokens really cost more than input tokens?

Yes, typically four to five times more across the major providers, with flagship tiers around $2 to $3 per million input against $10 to $15 per million output. That asymmetry means answer length is a pricing decision rather than a style one, and that removing 100 tokens from a response is worth roughly 400 to 500 removed from a prompt. It is the most commonly overlooked fact in token optimization.

How do I measure token efficiency properly?

Track tokens per completed task rather than per call. Tokens per call is misleading because it improves when quality degrades: a prompt trimmed too far causes more failures, more retries, and a lower average per call while the total bill rises. Counting every attempt that contributed to one finished outcome exposes that immediately, which is why it is the only metric that cannot flatter a bad change.

What are reasoning tokens and do I pay for them?

The internal thinking a reasoning model generates before its answer: plans, checks and discarded attempts that never appear in the response. They bill as output tokens, at output rates, and reasoning workloads consume 5x to 30x more tokens per task than plain completions. Control them with the effort setting and by keeping routine work off reasoning models.

What is Code Mode and how much does it save?

A pattern where the model writes code that calls your API instead of choosing from enumerated tool schemas. Cloudflare measured 81% fewer tokens on a multi-step task, and about 99.9% on an MCP server with 2,500 endpoints, where 1.17 million tokens of tool definitions collapsed to roughly 1,000. It needs a sandbox to run the code, and it pays off in proportion to how many tools you had.

Does JSON cost more tokens than CSV?

Yes, materially, for tabular data. JSON repeats every key name on every row and spends tokens on punctuation, where CSV and TSV carry the same values in roughly 30% to 60% fewer tokens. Keep JSON for genuinely nested structures, and use a tabular format when you are passing rows of records into a prompt.

Do unused tools cost tokens?

Yes, and it surprises people. The model reads every tool definition you pass in order to choose between them, so the full schema of every unused tool is billed as input on that call. In an agent loop you pay for them on every step and again on every retry. Filtering the tool list per step is usually the largest single cut in a tool-heavy system.

How do I reduce token usage in Claude Code or Copilot?

Mostly by scoping tasks rather than shortening prompts. The context in a coding agent is the code it has read, and it accumulates across turns, so start fresh sessions at task boundaries. Keep whatever sits at the top of the prompt stable so caching keeps working, and do not ask the most capable model for renames and formatting.

Will token optimization fix a high AI bill on its own?

Only partly. Token optimization reduces the size of each request but not the number of requests, so it cannot touch an agent looping more than it should or an unlogged retry storm. Fix call volume first, then request size. Applied in that order the savings compound; applied in reverse you are making the wrong number smaller.

Sources

• "Token optimization: The backbone of effective prompt engineering," IBM Developer
• "LLM Token Optimization: Cut Costs and Latency in 2026," Redis
• "Towards Optimizing the Costs of LLM Usage" (arXiv, 2024)