Context Rot: Why More Tokens Make Your LLM Worse

TL;DR

Context rot is the decline in a language model's performance as its input grows, on the same task, well inside the advertised window. Chroma measured it across 18 models in 2025, from GPT-4.1 and Claude 4 to Gemini 2.5 and Qwen3, and every single one degraded as input length increased. Not one was immune.

The uncomfortable part is why nobody noticed sooner: the standard long-context benchmark, needle in a haystack, is too easy. Models ace it while failing the version that resembles real work, where the phrasing of the question does not match the phrasing of the answer and the document is full of almost-right distractors.

Rot is invisible on your invoice and expensive anyway. It does not raise the price of a call; it lowers the success rate, and cost per completed task divides by exactly that number. A team watching cost per call sees nothing while the retries pile up.

This is the failure-mode post of the context engineering series. It covers what rot is, how it was measured, why it happens, what it costs, and how to test your own system for it. The fixes, compression, memory, retrieval, each get their own post.

Running agents whose context grows every step? We design AI agents with token budgets and rot testing built in from the first sprint, because the failure shows up in the success rate, not the invoice.

Key Takeaways

• Chroma evaluated 18 LLMs across three task families. Performance consistently worsened as input length increased, on every model, including the frontier ones.
• Classic needle-in-a-haystack scores hide the problem: retrieval of a distinctive fact stays strong while semantically harder versions of the same task collapse.
• A single distractor measurably reduced accuracy. Claude models hallucinated least under pressure, GPT models most.
• Shuffled haystacks outperformed logically ordered ones, the opposite of what anyone would design for.
• The same 306 questions answered from a focused ~300-token input versus the full ~113K-token history showed a substantial gap across every model family.
• Rot converts directly into money through the retry path: a success rate falling from 95% to 70% raises cost per completed task by roughly 36% before anyone notices a symptom.

What is context rot?

Context rot is the measurable decline in a language model's performance as the amount of input it processes grows, even when the task itself stays the same. The term comes from a 2025 Chroma research report that evaluated 18 models, including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3, across tasks ranging from simple retrieval to conversational question answering. Every model tested degraded as input length increased; long-context benchmarks had hidden the effect, because retrieving one distinctive fact is far easier than reasoning over ambiguous, distractor-filled material. Rot is not the same as running out of window. It happens well inside the advertised limit, on inputs the model nominally supports, and it compounds in agents, because every step appends to the context that the next step must survive. The practical consequence is blunt: input size is a quality variable, not just a capacity or cost variable.

How did Chroma actually measure it?

The Chroma report is the closest thing this failure mode has to a definitive measurement, and its design is what makes the findings hard to dismiss. Three task families, 18 models, input length as the only variable being stretched.

Extended needle in a haystack. The classic test hides a distinctive sentence in a long document and asks for it back. Chroma kept the shape but varied what the classic version holds constant: how semantically close the question is to the needle, whether distractors that almost answer the question sit nearby, and how the surrounding text is structured.

LongMemEval. Conversational question answering over chat history, 306 questions, with prompts around 113K tokens. The same questions were also asked against a focused input of roughly 300 tokens containing just the relevant material.

Repeated words. A stress test with no reasoning at all: reproduce a sequence with one unique word inserted, across 1,090 context length variations. Even this degraded with length.

Why does the standard benchmark miss it?

Because finding a distinctive needle is a lexical task, and production work is a semantic one. When the question shares phrasing with the target sentence, attention has an easy anchor, and models stay near-perfect deep into six-figure token counts. Marketing charts are built from this version.

Weaken the anchor, ask the question in words that do not appear in the document, and add material that almost answers it, and the same models start failing at lengths their benchmark scores said were safe. Chroma quantified the gap directly: needle-question similarity in their corpora ranged widely, and the lower the similarity, the faster performance fell as input grew.

One more result deserves its own sentence, because it breaks an assumption almost everyone holds: haystacks with their sentences shuffled into random order produced better retrieval than the same content in its original, logically flowing form. The narrative structure you preserve for human readers is not helping the model find anything.

Why does context rot happen?

The mechanism is the attention budget the series hub is built around. Transformers spread attention across every token through a softmax, so each added token dilutes the weight available to all the others. The Stanford lost-in-the-middle study showed where the dilution lands: recall is strong for material at the beginning and end of the window and drops by roughly twenty points for the same material placed in the middle, with the trough deepening as context grows.

Position is only half of it. Anthropic's engineering guide names the other half: models have less training exposure to very long sequences, so the further an input stretches past the lengths the model saw most, the less reliable its handling becomes. Rot is what those two effects look like from the outside, gradual, uneven, and invisible until you measure for it.

What does rot look like in production?

You have probably already seen it and filed it under something else.

• The support bot that contradicts a constraint the customer stated fifteen turns earlier.
• The coding agent that re-reads a file it already has in context, because it can no longer find the contents it is holding.
• The document QA system that answers perfectly from page one and page forty, and invents things about page twenty.
• The agent run that gets less precise the longer it runs, even though every individual step looks reasonable in the trace.

None of these throw an error. The window was never exceeded, so nothing logged. The only signature is a success rate that drifts down as sessions grow long, which is exactly why teams that only monitor per-call latency and cost never see it.

What does context rot cost?

Here is the bridge to the cost series, and the reason this post exists on this site rather than as one more summary of Chroma's charts.

Rot does not change the price of a call. It changes the probability the call did the job. The unit that captures that is cost per completed task: what you paid per attempt, divided by the success rate. Divide by a shrinking number and the curve does the rest.

Agents make it worse twice over. Every step appends to the context, so late steps run in exactly the rotted conditions where mistakes are most likely, and every retry of a failed step re-sends the entire accumulated window, which is the mechanism behind agent runs costing 19x to 50x a single call. Rot pushes the failure rate up at precisely the point where each failure is most expensive to repeat.

How do you test your own system for rot?

You do not need a research lab. You need one honest comparison, run on your own data, and Chroma's LongMemEval design hands you the template.

Take twenty real questions your system has answered in long sessions. Answer each twice with the same model: once against the full accumulated context, once against a focused input holding only the material a colleague would deem relevant, a few hundred tokens. Score both sides blind.

If the focused side wins, and in Chroma's testing it won across every model family, you now have a number: how much accuracy your accumulated context is costing you, on your workload, this month. That number is the business case for every technique in the rest of this series, and it took an afternoon.

Where the fixes live

Deliberately brief, because each fix is its own post in the series. Ordering puts critical material at the edges, where the lost-in-the-middle curve says recall survives. Compression stops the window growing by summarising what accumulated. Memory moves knowledge out of the window entirely and reads it back on demand. Retrieval, the biggest lever, replaces the full history with the focused slice that just beat it in your test, and its economics are already priced in the cost series.

What none of the fixes change is the constraint itself. Rot is a property of the architecture, all 18 models, remember, so the discipline is not choosing a model that does not rot. It is engineering the window so rot has less to work with.

Frequently asked questions

What is context rot in AI?

Context rot is the decline in a language model's accuracy as its input grows, even when the task stays identical and the input stays inside the advertised context window. Chroma's 2025 research measured it across 18 models from four families, and every model degraded as length increased, including frontier ones.

Is context rot the same as lost in the middle?

They are related but not identical. Lost in the middle describes where recall fails: material in the middle of a window is retrieved worse than material at the edges. Context rot is the broader effect: overall performance degrading as input grows, of which positional weakness is one visible symptom.

Why do models pass needle-in-a-haystack tests but still degrade in production?

The classic test hides a distinctive sentence and asks for it back with matching phrasing, which gives attention an easy lexical anchor. Production queries rarely quote their answers. When Chroma lowered the semantic similarity between question and needle and added distractors, performance fell at lengths the classic test had certified as safe.

Do bigger context windows fix context rot?

No. Rot occurs well inside advertised limits and appeared in every long-window model Chroma tested, so buying a larger window mostly buys more room for the effect to develop. Larger windows raise the ceiling on what you can send; they do not raise the reliability of what the model does with it.

How do I detect context rot in my own application?

Run a focused-versus-full comparison on your own data. Answer the same twenty real questions twice, once with the full accumulated context and once with a few hundred tokens of hand-picked relevant material, and score both blind. A gap in favour of the focused input is rot, measured on your workload.

Sources

• Chroma Research: Context Rot, how increasing input tokens impacts LLM performance
• Liu et al.: Lost in the Middle, how language models use long contexts
• Anthropic: Effective context engineering for AI agents
• Kamradt: the original needle in a haystack test