Context Engineering: Why the Model Is Not the Bottleneck

Context engineering explained: the attention budget, why long context fails, the levers that manage it, and what each one costs to run. With real numbers.

TL;DR

Context engineering is the discipline of deciding what a language model sees at inference time: which instructions, documents, history, memory and tool results enter the window, in what order, and at what moment. It absorbed prompt engineering the way architecture absorbs bricklaying. The prompt is one input among many, and in a modern agent it is usually the smallest one.

The reason the discipline exists is uncomfortable: more context makes models worse. Attention is a finite budget spread across every token in the window, recall is strong at the edges and weak in the middle, and every token is billed. So the job is not to give the model everything. It is to give it the least it needs to succeed.

This post is the map of that job. The budget you are spending, the two ways it fails, the levers that manage it, the retrieval decision that feeds it, the protocol that delivers it, and the packaging that makes it reusable. Each area gets its own deep dive in this series; this page tells you how they fit together and what each one costs, because context is not just an accuracy problem. It is the line item we spent twelve posts measuring in the LLM cost series.

Building agents that have to survive their own context window? We design AI agents around token budgets and cost per completed task, not vibes. Context architecture is the first thing we draw, not the last.

Key Takeaways

The model stopped being the limiting factor for most applications. What you put in front of it is.
Attention is a budget. Windows grew from 4K to a million-plus tokens in three years, but the ability to use tokens did not scale with the ability to hold them.
Recall is U-shaped: facts at the start and end of a window are retrieved far better than facts in the middle, which is where retrieval pipelines usually dump documents.
Context is the cost driver. The same question answered from a full 150K-token history costs roughly 19x what it costs from a retrieved 8K slice.
An agent re-sends its context on every step, so context bloat compounds quadratically in loops.
MCP standardises how context reaches the model; skills package it for reuse. Both are context engineering wearing infrastructure clothes.

What is context engineering?

Context engineering is the practice of designing what a language model sees at inference time: the system prompt, tool definitions, retrieved documents, conversation history, memory and state that surround the actual question. The term was popularised in June 2025, when Shopify CEO Tobi Lütke described it as providing all the context for a task to be plausibly solvable, and Anthropic's engineering team later formalised it as curating the smallest possible set of high-signal tokens. It differs from prompt engineering in scope: the prompt is one input among many, so prompt engineering became a subset of the larger discipline. The job exists because model quality stopped being the limiting factor for most applications. Attention is a finite budget, performance degrades as the window fills, and every token costs money, so deciding what enters the window, in what order, at what moment, is now the work.

That definition hides a shift in who does the deciding. In a chatbot, a human types the context. In an agent, the system assembles it: tool results stream in, history accumulates, retrieval fires, memory gets read. Nobody is curating that window by hand at step forty. The curation has to be designed in advance, which is exactly why this became an engineering discipline with its own failure modes and its own budget, rather than a knack for phrasing requests.

An illustrative mid-session agent window. The proportions shift by application; the imbalance does not.

Why did the bottleneck move from the model to the context?

Three years ago the window was the constraint. GPT-3.5 held 4K tokens, and the craft was squeezing a task into them. Then the ceiling lifted: 128K became table stakes, Claude models run 200K as standard, and million-token windows moved from research demo to priced product tier. The squeeze ended.

What did not scale is the model's ability to use what it holds. Anthropic's engineering guide describes an attention budget: transformers spread attention across every token in the window through a softmax, so each new token dilutes the weight available to the rest. Long before a window is full, the marginal token stops adding signal and starts subtracting it.

So the constraint inverted. When the window was small, engineering meant fitting things in. Now that it is huge, engineering means keeping things out. Gartner frames the same shift for enterprises: the differentiator is no longer access to a frontier model, which everyone has, but the pipeline that decides what the model sees. That pipeline is what this series takes apart.

Why does more context make a model worse?

Because recall is not flat across the window. The Stanford lost-in-the-middle study measured what happens when the answer to a question sits at different positions in a long context: models retrieve facts placed at the beginning or end far more reliably than the same facts placed in the middle, and the dip deepens as the context grows. Chroma's research ran the industry's models across increasing input lengths and found performance degrading with length even on trivially simple tasks, a phenomenon they named context rot. It is not an edge case. It is the default behaviour of every model they tested.

The U is the argument for ordering: position is not a formatting detail, it is retrieval probability.

Rot is the passive failure: quality decaying as tokens accumulate. There is an active one too. A hallucinated tool result, a stale summary or a hostile instruction that enters the window contaminates every turn that follows, because the model treats its context as ground truth. The failure branch of this series covers both: context rot in depth, and poisoning as its adversarial sibling.

What are the levers of context engineering?

Once you accept that the window is a budget with failure modes, the techniques organise themselves into a short list. Each gets a full post in this series; here is the map.

Lever	What it does	The trade it makes
Ordering	Puts critical material at the edges of the window, where recall is strongest	Free, but only helps what you kept
Compression	Summarises or compacts history so the window stops growing	Saves tokens; can delete the fact you needed later
Memory	Moves knowledge out of the window into storage, read back on demand	Unbounded capacity; every read is a paid retrieval
Retrieval	Selects small relevant slices instead of shipping whole corpora	The biggest lever, and a whole discipline of its own
Caching	Bills repeated prefixes at a fraction of full price	Cheapest win available; changes nothing about quality

Compression is what your agent framework calls compaction or auto-compact: when the conversation approaches the limit, an LLM summarises it and the summary replaces the transcript. It works until the summary silently drops the constraint the user stated forty turns ago.

Memory is the fashionable one, and the most oversold. Working memory IS the context window; everything else, vector stores, memory files, knowledge graphs, is external storage plus a read policy. Whether you need a memory product or a folder of notes depends on the task, and the memory post in this series is the buyer's guide the vendors will not write.

Ordering is free and almost nobody does it deliberately. The U-curve above is an instruction: system prompt and constraints at the top, the question and the freshest material at the bottom, bulk in the middle only when it must be there at all.

And underneath all of them sits the line that gives this series its first-principles post: prompt engineering optimises the wording of one input. Context engineering decides which inputs exist. Confuse the two and you polish sentences while the window fills with garbage.

RAG, fine-tuning or long context: how do you feed the window?

Every knowledge-heavy application faces the same fork: stuff the documents into the window, retrieve slices per query, or train knowledge into the weights. The honest answer is a decision framework, not a winner, and it runs on three variables: corpus size, change rate, and how much of the corpus each query actually touches.

Long context suits small, stable corpora hit by repetitive queries, where prompt caching turns the re-sent bulk into a 90% discount. Retrieval wins when the corpus is large, hot, or sparsely relevant, which is most production systems, and we have already priced its components line by line. Fine-tuning teaches behaviour, not facts, and is the right answer far less often than its invoice suggests. The retrieval branch of this series works the decision in full, including the version where the agent drives its own searches and the bill compounds accordingly.

What does context actually cost?

This is the part of the discipline most writing skips, and it is the reason this series exists on this site. Context is not just an accuracy budget. It is the number on the invoice.

Input tokens are the quiet majority of most LLM bills. At Claude Sonnet's list input price of $3 per million tokens, a single call that carries 150K tokens of accumulated history costs about $0.45 before the model writes a word. The same question answered from a compacted 40K window costs about $0.12. From a retrieved 8K slice, under three cents. Same model, same question, a 19x spread decided entirely by context strategy.

Illustrative arithmetic at list price, before caching. Caching narrows the gap; it does not close it.

Agents multiply this. Every step of a loop re-sends the accumulated context, which is why an agent run costs 19x to 50x a single call and why input grows with the square of the step count. Context engineering and cost engineering are the same discipline viewed from different dashboards, and the token-level version of this argument is already written.

How does MCP change context delivery?

Everything above assumes the context can reach the model. The Model Context Protocol is the plumbing that makes that a standard instead of a custom job. Introduced by Anthropic in November 2024 and since adopted across the major tooling ecosystems, MCP defines how an AI application discovers and calls external tools, resources and prompts.

The pre-MCP world was an integration matrix: every application times every system, each pair its own connector. The protocol collapses the matrix, each side implements the standard once.

The multiplication is why the standard won. The addition is what you maintain instead.

Two honest caveats belong on this map. First, MCP is not always the answer: for one application talking to one internal system, a plain API is less machinery, and the comparison post in this series draws that line without protocol romance. Second, exposing tools to a model is a security surface. The spec has mandated OAuth 2.1 for remote servers since March 2025, and we build to it; the production guide to building an MCP server that survives day two is next in this series, and it is where the plumbing gets its hands dirty.

What are agent skills, and why do they belong on this map?

A skill is a folder with a SKILL.md file: instructions, scripts and resources an agent loads only when the task calls for them. The mechanism that matters is progressive disclosure. The agent holds one line of metadata per skill until one becomes relevant, then loads the full instructions, then any bundled files. Context arrives just in time instead of squatting in the window from turn one.

That is why skills close this series rather than opening it: they are the packaging layer of everything above. A well-built skill is a compression decision, an ordering decision and a retrieval policy shipped as a folder. We build them as client work, and the skills post covers the part the official docs leave out: when a skill beats an MCP server, and when it is just a README with ambitions.

What does Anthropic's context engineering guide actually say?

Anthropic's effective context engineering for AI agents is the closest thing the discipline has to a canonical text, and most summaries of it stop at the headline. The working content is four positions worth having in your head.

Context is a finite resource with diminishing returns. The guide's attention-budget framing, which this post borrowed above: every token depletes a budget, so the goal is the smallest set of high-signal tokens that makes the desired outcome likely.

System prompts should sit at the right altitude. Too specific and you have brittle if-else logic in prose; too vague and the model improvises. The guide's advice is strong heuristics, minimal fixed logic, and no laundry lists of edge cases.

Prefer just-in-time context over pre-loading. Give the agent lightweight references, file paths, queries, links, and let it pull the material when needed, the way a person uses a filing system instead of memorising the cabinet.

For long horizons: compact, take notes, or delegate. Summarise when history bloats, persist critical state outside the window, and hand focused sub-tasks to sub-agents that return distilled results instead of their whole transcripts.

None of this requires Anthropic's stack. It is a description of working within a budget, and every branch of this series is one of those four positions taken seriously enough to measure.

Frequently asked questions

What is context engineering in simple terms?

It is deciding what a language model gets to see when it answers: the instructions, documents, history, and tool results that surround the question. Good context engineering gives the model the least material it needs to succeed, because attention is limited, extra tokens degrade recall, and every token is billed.

Is context engineering replacing prompt engineering?

It absorbed it rather than replaced it. The prompt is one input among several that reach the model, so wording instructions well is now a subset of the larger job: choosing which documents, history, memory and tool results enter the window at all, in what order, and at what moment.

Why do models get worse with more context?

Attention is a fixed budget spread across every token in the window. As input grows, each token receives less of it, and recall becomes uneven: models retrieve facts placed at the beginning or end far better than facts buried in the middle. Chroma's research measured this degradation across the major models and named it context rot.

Does a million-token context window remove the need for RAG?

No. Long windows and retrieval solve different constraints. A large window helps when most queries genuinely need to see a large, stable document set, and prompt caching can make that affordable. Retrieval wins when the corpus is big, changes often, or only small slices are relevant per query, which describes most production systems.

What is the Model Context Protocol (MCP)?

MCP is an open standard, introduced by Anthropic in November 2024, that defines how AI applications connect to external tools and data. Instead of writing a custom integration for every app-to-system pair, each side implements the protocol once. Since March 2025 the specification has required OAuth 2.1 for remote servers.