Prompt Caching: How Much It Saves (and When It Costs You)

TL;DR

Prompt caching stores the model's internal computation for the fixed prefix of your prompt, the system instructions, tool definitions and reference documents that repeat on every call, so you stop paying full price to re-process tokens the model has already seen. Cached tokens bill at roughly 10% of the standard input price across the major providers, a 90% discount on the repeated portion, and time-to-first-token drops by up to 80%.

The catch: some providers charge a write premium (Anthropic and GPT-5.6 add 1.25x on a cache write, 2x for a one-hour cache), so caching only pays off once enough requests read the cache to clear that premium, break-even lands around a 22% hit rate. Below it, caching costs more than not caching.

This guide covers what prompt caching is, exactly what each provider charges, the break-even math, when it backfires, and the one mistake that quietly drops your hit rate to zero. There's a prompt caching calculator to size the saving on your own numbers.

Paying frontier prices for tokens you've already sent? Our AI development work starts by instrumenting where the spend actually goes.

Key Takeaways

• Prompt caching bills the repeated prefix of a prompt at about 10% of the standard input price.
• The prefix must be byte-identical across requests to hit the cache; the variable user message sits at the end and is never cached.
• Providers with a write premium (Anthropic 1.25x, GPT-5.6 1.25x) need roughly a 22% hit rate to break even; DeepSeek and Gemini implicit caching save from the first read.
• The default cache lifetime is short, 5 minutes on Anthropic, so infrequent or slow traffic re-pays the write premium every call.
• The most common failure is a timestamp or session ID buried in the prefix, which changes the hash and forces a fresh write on every request.
• Real production teams report 49% to 80% token-cost reductions with caching applied correctly.

What is prompt caching?

When you send a prompt to an LLM, the model processes every token through its attention layers, producing an internal representation (the key-value, or KV, tensors) of your input. That computation is most of the cost and latency of a request. Prompt caching stores those KV tensors for a stable prompt prefix, so when a later request starts with the same prefix, the model skips the re-computation and picks up where it left off, like a bookmark, instead of re-reading from page one every time.

The output is unchanged. The model still generates a fresh, non-deterministic response; caching only affects the processing of input tokens you have already sent before. Two identical requests with cached prefixes will still produce different outputs.

That distinction matters, because prompt caching is easy to confuse with things it is not.

How prompt caching works

Every major provider implements the same core idea with small differences:

1. You mark (or the provider auto-detects) a cacheable prefix. On OpenAI and Gemini implicit caching this is automatic for prompts over ~1,024 tokens; on Anthropic and Bedrock you place explicit cache_control breakpoints.
2. First request, a cache write. The model processes the full prompt and stores the KV tensors for the prefix, keyed by a hash of its contents.
3. Later requests, a cache read. If the prefix hash matches, the model reads the stored tensors and only processes the new tokens at the end. Those cached tokens bill at the discounted read rate.
4. Expiry. Caches are ephemeral. After the time-to-live (TTL) elapses with no hits, the entry is dropped and the next request pays to write it again.

The rule that governs all of it: cache hits require an exact prefix match. Change one token anywhere before your breakpoint and the hash changes, the cache misses, and you pay to re-process everything.

What prompt caching is NOT

Misconceptions cause more wasted spend than ignorance does. Prompt caching is:

• Not response caching. It does not store and replay a previous answer. It stores internal computation; the model generates a fresh response every time.
• Not semantic caching. It does not match "close enough" prompts by meaning. It requires an exact prefix match, one changed character misses.
• Not fine-tuning. No weights are modified. It is a pure inference optimization; the model behaves identically whether the cache hits or misses.
• Not persistent. Caches expire in minutes to an hour. No traffic, no cache.
• Not automatic savings. Below a provider's minimum token threshold (usually 1,024) caching does not activate at all, and with a write premium it can cost more than it saves.

Prompt caching pricing, by provider

Read pricing is remarkably consistent, about a 90% discount, but the write premium and the defaults differ, and those differences decide whether caching helps you.

Anthropic's own published multipliers make the structure concrete: a 5-minute cache write costs 1.25x the base input price, a 1-hour write costs 2x, and every read costs 0.1x. On Claude Sonnet at $3 per million input tokens, that is $3.75 to write and $0.30 to read.

Does prompt caching actually save money?

Usually a lot, but not always, and the difference is the write premium.

With no write fee (DeepSeek, older OpenAI, Gemini implicit), every read is pure savings and caching wins from the first hit. With a write premium, you need enough reads per write to clear it. The break-even math for a provider that charges 1.25x on writes and 0.1x on reads:

• Uncached, each request costs 1x the prefix price.
• Cached, at hit rate h, each request averages (1 – h) x 1.25 + h x 0.1.
• Those are equal at h ~ 22%. Above a 22% hit rate you save; below it you lose.

In practice most production workloads sit far above break-even, a chatbot or agent reusing the same system prompt hits the cache on nearly every turn. Real reported results: Thomson Reuters Labs cut cost 60% on classification workloads; LangChain's Deep Agents measured 49% to 80% reductions across Claude, GPT and Gemini on real agent trajectories. The saving grows the longer a conversation runs, because the cached prefix is reused on every turn.

Try it on your own numbers. Enter your prefix size, request rate and cache TTL below. It works out your reads per write, the monthly saving, and whether caching is even worth it at your traffic, for Anthropic, OpenAI, Gemini and DeepSeek.

When prompt caching does NOT help

Caching is not free money. It actively costs more in four cases:

• Highly dynamic prompts. If the whole prompt changes each request, hit rate is zero and, on a write-premium provider, you pay 1.25x on every call with no reads to offset it.
• Infrequent requests. Caches expire in 5 minutes (default). If your calls are further apart than the TTL, the cache is gone before the next one arrives, you pay the write every time and never collect a read.
• Short prompts. Below the ~1,024-token minimum, caching does not activate. There is nothing to optimize.
• One-shot interactions. A single request with no follow-up is one write and zero reads, a net loss where there is a write premium.

The honest rule of thumb: if more than half your input tokens repeat across requests, caching should be on. If you cannot say which parts of your prompt are stable, fix that before touching caching settings.

The TTL trap: why you might still be paying full price

The single most common way teams think they are caching but are not: the prefix changed, or the cache expired, and nobody noticed.

Two culprits dominate:

A dynamic element buried in the prefix. A timestamp, session ID, or rotating instruction sitting before your user message changes the prefix hash on every request. The cache never matches. You pay a fresh write each call and never see a read. The fix is structural, not a settings change: put every stable thing first (tools -> system prompt -> reference docs), and every variable thing, the timestamp, the user's message, last.

The TTL running out between calls. Anthropic's default cache lives 5 minutes, refreshed for free on each hit. In tools like Claude Code that means if you step away for more than five minutes, the whole conversation re-caches at full price on your next turn. If your calls are naturally spread out, switch to the 1-hour TTL (2x write, but worth it when reads are more than five minutes apart) or keep the cache warm.

This is why cache hit rate is the metric to watch, not whether caching is "enabled". Log cache_read_input_tokens versus cache_creation_input_tokens and you will see immediately whether your prefix is actually stable.

How to implement prompt caching

The prompt structure is the same on every provider, static content first, dynamic content last:

[Tool / function definitions]   <- cached
[System prompt]                 <- cached
[Reference documents]           <- cached
[Conversation history]          <- cached (older turns)
[User query]                    <- always recomputed

• OpenAI, no code changes. Ensure the prompt exceeds 1,024 tokens and check cached_tokens in the response usage.
• Anthropic, add cache_control: {"type": "ephemeral"} on the last static block (up to 4 breakpoints); add "ttl": "1h" for the one-hour cache. One well-placed breakpoint is usually enough, the API looks back up to 20 blocks for a match.
• Amazon Bedrock, place cachePoint checkpoints in the system, messages or tools fields; per-model token minimums apply.
• Multi-provider, route through a gateway (LiteLLM, Portkey) that normalizes caching across providers rather than writing per-provider code.

One security note worth knowing: cached KV tensors live on provider GPUs, and researchers have shown timing side-channels can leak whether a prefix was cached. For anything beyond public data, read your provider's data-residency terms before enabling it.

Where prompt caching fits

Prompt caching is one of three levers for cutting a model bill, the others are routing work to smaller models and deleting the retries and oversized contexts nobody measures. Caching is the fastest to ship and often the highest ROI, but it is a tactic inside a strategy. For the full playbook, see our guide to LLM cost optimization.

Want the saving captured, not just calculated? We build the caching, routing and cost instrumentation into your stack end to end. Tell us what your invoice looks like.

Frequently asked questions

What is prompt caching?

Prompt caching is an inference optimization that stores the model's internal computation (the KV tensors) for the fixed prefix of a prompt, system instructions, tool definitions and reference documents that repeat across requests, so those tokens are not reprocessed and are billed at a steep discount. The model still generates a fresh response each time; only the cost and latency of the repeated input drop.

How does prompt caching work?

On the first request the provider processes the full prompt and writes the prefix's KV tensors to a cache, keyed by a hash of the prefix. On later requests, if the start of the prompt matches that hash, the model reads the cached tensors and only processes the new tokens at the end. Cached tokens bill at roughly 10% of the standard input price. The cache expires after a short time-to-live unless it keeps getting hit.

How much does prompt caching save?

Cache reads cost about 90% less than standard input tokens across the major providers, so a fully cached prefix costs roughly a tenth of its uncached price. The realized saving depends on your cache hit rate and whether your provider charges a write premium; production teams commonly report 50% to 80% reductions on the repeated portion of their prompts.

Does prompt caching save money?

It does whenever a meaningful share of your requests reuse the same prefix. Providers with no write fee (DeepSeek, older OpenAI, Gemini implicit) save from the first cache hit. Providers with a write premium (Anthropic, GPT-5.6 at 1.25x) break even around a 22% hit rate, above that they save, below it caching costs more than not caching.

What can be cached in a prompt?

The stable content at the start of a request: tool and function definitions, the system prompt, few-shot examples, large reference or RAG documents, and older conversation turns. The variable user query must sit at the end and is never cached. On most providers the cacheable prefix must be at least 1,024 tokens.

Why is my prompt cache not working?

Almost always because the prefix is not byte-identical between requests. A timestamp, session ID or reordered field before your breakpoint changes the hash and forces a fresh write every call. The other cause is expiry: the default cache lives about five minutes, so calls spaced further apart than the TTL never hit a warm cache. Move all dynamic content to the end of the prompt, and use the 1-hour TTL for slow traffic.

What is the difference between prompt caching and semantic caching?

Prompt caching skips recomputation of an exact, repeated prompt prefix and always returns a fresh model response. Semantic caching matches a new query to a similar previous one by embedding distance and returns the stored answer, skipping the model entirely, which is cheaper but risks stale or wrong responses. They solve different problems and stack well: semantic caching catches repeated questions, prompt caching reduces cost on everything else.

Sources

• Prompt caching, Claude Platform Docs (Anthropic)
• Prompt caching, OpenAI API guide
• Prompt caching for faster model inference, Amazon Bedrock
• Context caching, Google Gemini API