Context Engineering for AI Agents: What It Is and Why It Matters

A 200K token context window sounds like plenty until you run journalctl -n 1200 on a busy server and hand the result to an agent. That one command measures 52,724 tokens, a quarter of the window, before the model has reasoned about anything. Do it twice and the agent is half full of logs it will mostly ignore.

Context engineering is the discipline that decides what goes into that window and what stays out. It absorbed prompt engineering rather than replacing it: the prompt is now one layer in a stack that also holds tool definitions, retrieved documents, message history, and every byte of tool output an agent has ever seen. Get the stack wrong and you hit context rot, where accuracy falls as the input grows even though nothing about the task got harder. Get it right and the same model on the same hardware finishes tasks it used to abandon.

This guide covers the distinction from prompt engineering, the four documented ways a long context fails, what real command output actually costs in tokens, and the strategies production teams use to keep agents inside their budget. Every token figure below was measured rather than estimated, tested August 2026 against the Claude token counting API with real repository files and real journald output from a Rocky Linux 9 web server.

Context engineering vs prompt engineering

Prompt engineering asks how to phrase an instruction. Context engineering asks what the model should be able to see when it reads that instruction. Anthropic frames the difference as a scope change: prompt engineering covers “writing and organizing LLM instructions”, while context engineering covers “the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference”, including everything that lands in the window from outside the prompt.

The term settled quickly. Shopify’s Tobi Lütke described it in June 2025 as the art of providing all the context needed for a task to be plausibly solvable, and Andrej Karpathy’s endorsement a week later pushed it into common use. The academic literature caught up fast: a 2025 survey of context engineering reviewed over 1,400 research papers and split the foundational components into context retrieval and generation, context processing, and context management, with RAG, memory systems, and multi-agent systems as the implementations built on top.

For a single-turn chatbot the distinction barely matters, because the prompt is most of the context. For an agent it matters enormously. An agent reasons across many steps, calls tools, retrieves files, and replays its whole history on every turn, so the prompt shrinks to a rounding error while everything around it grows. Anyone who has watched a terminal coding agent work through a refactor has seen this: the instruction was one line, and the window filled with file reads.

What sits in the context window

Six things occupy the window on every request, and only one of them is what the user typed: the system prompt, the tool definitions, any retrieved documents, the message history, the results of every tool call so far, and the query itself.

Tool definitions are the layer engineers forget, because they never appear in a transcript. They are sent as JSON schemas on every single request. To find the real cost, define a plausible DevOps tool set (kubectl, Helm, Terraform, git, Docker, Prometheus, Loki, AWS, filesystem) and count tokens as tools are added:

curl -s https://api.anthropic.com/v1/messages/count_tokens \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{"model":"claude-sonnet-4-5-20250929","messages":[{"role":"user","content":"hi"}],"tools":[]}'

The endpoint bills nothing, though it is rate limited by usage tier, and returns the exact input size the model would see:

{"input_tokens":8}

Adding the tool schemas one group at a time produces a curve worth knowing:

That fixed cost is worth isolating. A single tool whose name, description, and schema are all empty still costs 529 tokens, because enabling tool use injects instructions of its own. After that, each realistic DevOps tool adds roughly 94 tokens. So a 32-tool agent spends about 3,600 tokens per request on capabilities it may never invoke, and an agent wired to several MCP servers can pass 10,000 tokens of schemas before the conversation starts. This catches people out when they add servers casually.

Why bigger context windows do not fix it

Transformer attention is the reason. Every token attends to every other token, which is n² pairwise relationships for n tokens, so a model’s ability to hold those relationships gets thinner as the window fills. Anthropic describes the result as an attention budget that every added token spends down.

Chroma tested this properly. Their context rot report ran 18 models (Claude Opus 4 and Sonnet 4, o3, GPT-4.1, Gemini 2.5 Pro and Flash, Qwen3 and others) on tasks engineered to hold difficulty constant while input length varied. Performance still fell in every family. Three findings stand out:

• Retrieval degrades faster when the question and the answer share little wording. The lower the similarity, the steeper the decline as input grows.
• A single distractor, one plausible but wrong passage, measurably lowers accuracy, and adding more compounds it.
• Models scored better on shuffled haystacks than on logically structured ones. Chroma flags this as counterintuitive and unexplained rather than as advice, and nobody should start shuffling their own documents over it.

The same report compared a focused prompt of roughly 300 relevant tokens against a full 113,000-token prompt on LongMemEval. Every model family lost ground on the full version. Reasoning modes lifted the scores on both, and the gap between them stayed. Position matters too: the Lost in the Middle work established the U-shaped curve where models recall the start and end of a context reliably and the middle poorly, including models explicitly built for long inputs.

None of that says long windows are useless. It says capacity and usable capacity are different numbers, and only the first one appears on a pricing page.

The four ways a long context fails

Drew Breunig’s taxonomy is the most useful vocabulary here, because each failure has a different fix. His two posts on how long contexts fail and how to fix them carry the supporting evidence.

Context poisoning

A hallucination lands in the context and then gets referenced as fact on every later turn. The documented case is a Gemini agent playing Pokémon: it invented game state, the false state stayed in history, and the agent built strategies toward goals that could not be reached. The fix is validation at the boundary, not better prompting, because the model has no way to distrust its own transcript.

Context distraction

The context grows until the model favours repeating its own history over reasoning from training. The same Gemini agent tipped past roughly 100K tokens and began recycling prior actions instead of forming new plans. Databricks measured correctness falling from around 32K tokens for Llama 3.1 405B, well inside the advertised window.

Context confusion

Irrelevant content gets used. Reading the Berkeley Function-Calling Leaderboard, Breunig points out that every tested model performs worse once more than one tool is offered. A quantized Llama 3.1 8B failed with 46 tools available and succeeded with 19. DeepSeek-v3 showed confusion above 30 tools and was close to guaranteed failure past 100. Selecting tools dynamically instead of loading everything improved that Llama model by 44%. Even in the runs where dynamic selection did not lift the score, it still cut power draw by 18% and response time by 77%.

Context clash

New information contradicts what is already in the window. Microsoft and Salesforce found that spreading a task across multiple conversational turns cost an average 39% in performance, with o3 dropping from 98.1 to 64.1. The cause is early wrong attempts staying in history and anchoring later reasoning.

What real command output costs in tokens

Token budgets are easier to respect when you know the exchange rate for the content your agents actually read. Measuring real files and real command output against the counting API gives a table you can plan against:

The spread is the useful part. Structured machine output costs 1.5 to 2 times as many tokens per kilobyte as prose or configuration, because timestamps, UUIDs, quoted keys, and punctuation all tokenize badly. A 200K window that holds most of a book holds under 400 KiB of JSON. Anyone who pipes raw API responses into a context is paying the worst rate on the board.

The log case is the one that bites in practice. On a production web server, 1,200 journal lines came to 52,724 tokens, about 44 tokens per line and 26% of a 200K window from one command. Filtering the same output to lines matching common failure keywords left 41 lines and 1,433 tokens:

journalctl --no-pager -n 1200 | grep -iE 'error|fail|denied|timeout|refused|fatal|panic'

That is a 97% reduction, 37 times fewer tokens, on the same underlying data. The filtered version is also more useful to the model, because it contains a higher share of what the question was about. Relevance and thrift point the same direction more often than people expect, which is why “give the model everything and let it decide” is the most expensive habit in agent design.

Write, select, compress, isolate

LangChain’s four-strategy framework maps cleanly onto what teams actually build, so it is worth using as a checklist.

Write means putting state outside the window so it survives. A scratchpad file, a plan the agent updates as it works, or a notes directory it can re-read. Manus, whose team published a detailed account of running agents in production, treats the filesystem as unlimited persistent context and keeps only references (paths and URLs) in the window itself.

Select means pulling in only what the current step needs. This is where retrieval earns its place, and reports of its death were premature. Whether you run a retrieval pipeline on Qdrant or a self-hosted stack on pgvector, the job is the same: return the few passages that matter rather than the corpus. Chunk size and how you configure the collection stop being storage details here and become context decisions, because they set how much irrelevant text rides along with each hit. Anthropic pairs this with just-in-time loading, where the agent holds lightweight identifiers and fetches content through tools when it needs it, the way an engineer keeps a file tree in mind instead of memorizing every file.

Compress means keeping the meaning and dropping the tokens. Summarizing finished sub-tasks, truncating tool results to the fields consumed, or pruning aggressively. The Provence pruning model cut a Wikipedia article by 95% while preserving what was needed to answer a specific question.

Isolate means splitting work so no single window carries everything. Sub-agents are the common form: each gets a clean window, spends tens of thousands of tokens, and returns a distilled summary of one or two thousand. Anthropic’s multi-agent research system beat single-agent Claude Opus 4 by 90.2% on their internal research evaluation using exactly this shape. If you use specialized subagents in Claude Code, that is context isolation with a configuration file.

The tool-loadout question sits in the same bucket. Instead of registering 32 tools permanently, scope the set to the task. Our measurements put 8 tools at 1,419 tokens against 3,578 for all 32, and the Berkeley results above suggest the smaller set also chooses better.

Managing context across a long task

Long-running agents need a policy, not a one-time layout. The window fills, gets trimmed, and refills, and the agent has to stay coherent across the seam.

Compaction summarizes a conversation approaching the limit and restarts with the summary in place. Anyone who has run /compact in a coding agent has used it, and the Claude Code command reference covers the manual trigger. The judgement call is aggression: trim too hard and you lose a detail whose importance only becomes clear later. Clearing stale tool results is the low-risk version, since their value usually expires the moment they are read.

This is now an API-level feature rather than something you build. Claude’s documentation currently points at server-side compaction as the primary strategy and treats the clearing policies below as the fine-grained option. Tool result clearing is configured with a beta header and a context_management block. The same header and block go on /v1/messages in a real agent loop; count_tokens is the free way to preview a policy before wiring it in:

curl -s https://api.anthropic.com/v1/messages/count_tokens \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "anthropic-beta: context-management-2025-06-27" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5-20250929",
    "messages": [{"role": "user", "content": "hi"}],
    "context_management": {
      "edits": [{
        "type": "clear_tool_uses_20250919",
        "trigger": {"type": "input_tokens", "value": 100000},
        "keep": {"type": "tool_uses", "value": 3},
        "clear_at_least": {"type": "input_tokens", "value": 5000},
        "exclude_tools": ["read_file"]
      }]
    }
  }'

The response confirms the policy attached and reports the pre-edit size:

{"input_tokens":8,"context_management":{"original_input_tokens":8}}

Note that the beta goes in the anthropic-beta header. The official SDKs take a betas argument instead, and passing that key in a raw HTTP body is rejected outright with betas: Extra inputs are not permitted, so a request that looks right in SDK examples fails against the bare API.

The defaults are sensible: clearing activates at 100,000 input tokens and keeps the last three tool interactions. The parameter people skip is clear_at_least, and it exists because clearing tool results invalidates the cached prompt prefix from the edit point forward. Every clear bills a cache write, so a policy that trims 500 tokens repeatedly costs more than it saves. Set a floor big enough to be worth the invalidation. Pointing exclude_tools at whichever tool returns your durable facts is the other detail worth getting right.

The payoff is measured rather than theoretical. On a 100-turn web search evaluation, context editing cut token use by 84% and let agents finish runs that previously died on context exhaustion. On a separate internal agentic-search evaluation, pairing context editing with the memory tool moved performance 39% above baseline, against 29% for context editing alone.

Structured note-taking is the complement. The agent writes findings to files as it goes, so a cleared window costs it nothing important. Manus does this with a running task file that it rewrites at each step, which has a second effect worth stealing: repeating the plan at the end of the context keeps the objective inside the model’s most reliable attention region and fights the middle-of-context blind spot.

Where the practitioners disagree

The field is young enough that credible teams contradict each other, and knowing where the disagreements are is more useful than picking a side early.

On sub-agents, Anthropic reports large gains from parallel isolation, while Cognition argues in Don’t Build Multi-Agents that the approach is unreliable because subagents act on partial information. Their two principles are that you should share full agent traces rather than individual messages, and that every action carries an implicit decision, so parallel agents making incompatible assumptions produce incompatible work. The reconciliation is that isolation works for read-only breadth (search, research, audit) and fails for writes that must agree with each other. Split a codebase migration across four agents and you get four incompatible interpretations.

On pruning, the instinct is to strip failures out of the context. Manus deliberately leaves them in, on the grounds that a model which can see its failed attempt will not repeat it, and that error recovery is what separates an agent from a script. That sits awkwardly beside the context-clash evidence, where stale wrong attempts drove a 39% average drop. The distinction that matters is whether the failure is labelled. A tool error with a clear message teaches. A half-finished wrong answer with no marker misleads.

On cost, Manus reports the metric most teams never instrument: KV-cache hit rate, which they call the single most important number for a production agent. Cached input tokens cost about a tenth of uncached ones, and their agents run roughly 100:1 input to output, so cache misses dominate the bill. The practical consequence is counterintuitive. Do not rebuild your system prompt per turn, do not inject a current timestamp near the top, and keep the history append-only, because any edit invalidates the cache from that point on. That is also why trimming is not free: clearing tool results rewrites the prefix and bills a cache write, which is exactly the tension clear_at_least exists to manage.

What to measure in your own agent

Four numbers turn this from theory into something you can act on, and none of them require new infrastructure.

Start with context utilization at the moment of failure. When an agent goes in circles, log how full the window was. If the pattern clusters somewhere well under the limit, you have found your real ceiling, and it is the one to design against rather than the advertised one.

Then measure tokens per completed task, split by layer. Tool schemas, retrieved documents, tool results, and history each have an owner and a fix. Agents use roughly four times the tokens of a chat interaction and multi-agent systems around fifteen times, so this number decides your unit economics before any model choice does.

KV-cache hit rate comes next, because it is usually the cheapest large win available. A prompt prefix that changes every turn can multiply cost by ten with no change in behaviour, and the fix is often deleting a timestamp.

Finally, track the ratio of relevant to total tokens in tool results. The journal example above ran at 1,433 useful tokens inside 52,724 delivered. Filtering at the tool boundary is the biggest single win most teams have not taken, and it costs one grep. In production, you’ll want that filter living in the tool itself rather than in a prompt asking the model to ignore things, because a model cannot unsee what you sent it.