Context Engineering Explained (2026): Why 1M Tokens Still Isn't Enough

Plug five MCP servers into a coding agent and it can get worse before you type a word. Nothing is broken: the tool definitions alone eat a large slice of the window, and every turn after that pays rent on them. I find that the cleanest argument for context engineering: the discipline of deciding which tokens an LLM sees at each step, rather than polishing the wording of one prompt. This post explains what context engineering means in 2026, why it displaced prompt engineering as the thing teams argue about, what the research on long contexts actually shows, and the five techniques (retrieval, tool design, memory, compaction and sub-agents) that do most of the work. It is written from the infrastructure side, because most of these problems look a lot like capacity planning.

What is context engineering, and why did it replace prompt engineering?

Anthropic's applied AI team published the clearest definition in Effective context engineering for AI agents on 29 September 2025: the set of strategies for curating and maintaining the optimal set of tokens during inference, including everything that lands in the window besides the prompt itself. That includes the system prompt, tool definitions, retrieved documents, the message history, tool results and any notes the agent wrote for itself. The term had been circulating since June 2025, when Andrej Karpathy argued for it over prompt engineering in a post on X on 25 June (summary of the exchange); Anthropic's post is where it became engineering guidance.

The shift happened because the workload changed. A single-turn chat completion is mostly the prompt, so wording mattered. An agent running for fifty turns is mostly accumulated state: file contents it read, test output, error logs, JSON from tools. By turn thirty, the carefully written system prompt is a few percent of the window. Anthropic's multi-agent research write-up from June 2025 puts numbers on the scale: agents typically use about four times the tokens of a chat interaction, and multi-agent systems about fifteen times. When most of the input is produced by the loop itself, the interesting question is not how to phrase an instruction but what to keep, what to drop and what to fetch later.

LangChain's Context Engineering for Agents (July 2025) gave the field a useful four-word taxonomy: write (save context outside the window), select (pull in what the step needs), compress (keep only the tokens that matter) and isolate (split work across separate windows). I use those four verbs as a checklist when reviewing an agent design, the same way I would ask about reads, writes, caching and partitioning for a database.

Context windows vs effective context: what the context rot research shows

Vendors advertise context windows of a million tokens. The research consistently says the usable part is smaller, and that degradation starts long before the limit.

Chroma's Context Rot report (July 2025, Kelly Hong, Anton Troynikov and Jeff Huber) tested 18 models from Anthropic, OpenAI, Google and Alibaba. Performance became less reliable as input length grew, even on simple tasks. A few findings stuck with me. When the question and the relevant passage shared fewer words and less meaning, accuracy fell off faster with length. A single distractor passage hurt; four hurt more. On a LongMemEval subset of 306 prompts, every model did significantly better on a focused prompt of about 300 tokens than on the full version of about 113k tokens containing the same answer. Strangest of all, haystacks of randomly shuffled sentences beat coherent documents across all 18 models, which suggests that structure in the filler pulls attention in ways nobody designed for.

The NoLiMa benchmark (ICML 2025) minimised literal word overlap between question and answer. Of 13 models claiming at least 128K context, 11 fell below half their short-context score by 32K tokens; GPT-4o dropped from 99.3 percent to 69.7 percent. Anthropic's own post explains the mechanism in plain terms: self-attention relates every token to every other, n squared pairs for n tokens, so each extra token draws down a finite attention budget.

The practical reading for an engineer: treat the advertised window like the maximum throughput on a disk spec sheet. You can hit it, but you should not plan to run there.

The token budget is mostly tool definitions and tool output

When I profile an agent's context, the prompt is rarely the problem. Tool definitions and tool results are. Anthropic's advanced tool use post (November 2025) gives a five-server example: 58 tools consuming about 55K tokens before the conversation starts.

The same post reports that loading tool definitions on demand cut upfront context from about 77K to 8.7K tokens and raised accuracy on Anthropic's internal MCP evaluations from 49 to 74 percent for Opus 4 and from 79.5 to 88.1 percent for Opus 4.5. Those are vendor numbers on vendor benchmarks, but the direction matches the context rot research: fewer, better-separated tools beat a large menu. Anthropic's tool guidance boils down to tools with minimal overlap that return token-efficient results. If a human engineer cannot say which of two tools to call, the model will not do better. I covered the protocol side in What Is MCP; the context cost is the part most teams find out about late.

Tool output is the other half. A kubectl get pods -A -o json on a busy cluster can run to tens of thousands of tokens, and the agent needed three fields. Truncating, paginating and summarising inside the tool, not after it, is the cheapest context engineering there is.

Retrieval: just-in-time loading beats stuffing the prompt

The old RAG pattern was to embed everything, retrieve the top chunks and paste them in before the model starts. Coding agents moved the other way. Anthropic describes Claude Code as a hybrid: CLAUDE.md files are loaded up front, then the agent uses glob and grep to fetch files at the moment it needs them, holding lightweight references (paths, queries, URLs) instead of contents. The agent explores the way an engineer does: list the directory, read the interesting file, ignore the rest.

This is not free. Just-in-time retrieval costs turns and latency, and an agent with bad search tools wanders. But it keeps the window aligned with the current step instead of with a guess made at turn zero. My rule of thumb: pre-load what every step needs (build commands, conventions, the shape of the repo), and make everything else fetchable. I went through the trade-off against long context and fine-tuning in RAG vs Long Context vs Fine-Tuning.

Static context files deserve a sceptical note. A paper by Thibaud Gloaguen and colleagues, Evaluating AGENTS.md (first posted February 2026, revised in September), found that repository context files generally did not improve task success and raised inference cost by over 20 percent across several agents and models. Agents did follow the instructions; repository overviews just did not help them find the right files. The authors conclude that such files are useful for specifying non-standard coding practices, and that anything beyond that should be evaluated before you rely on it. That matches my experience: a context file should hold what the agent cannot discover by reading the code.

Memory, compaction and context editing for long-running agents

Long tasks overflow any window, so something has to give. There are three mechanisms, and they compose.

Compaction summarises a conversation near its limit and restarts with the summary. Anthropic's post notes that Claude Code keeps the five most recently accessed files alongside the summary. The hard part is choosing what to discard: an over-eager summary drops the one detail that matters ten turns later.

Context editing is the lightest touch: clear old tool results that will never be read again, keep the conversation. Anthropic's context management announcement (29 September 2025) reported that on a 100-turn web search evaluation, context editing let agents finish workflows that otherwise died of context exhaustion, while cutting token use by 84 percent.

Memory writes notes outside the window and reads them back later: a progress file, a to-do list, a NOTES.md. The same announcement reported a 39 percent improvement on an internal agentic search evaluation when the memory tool was combined with context editing, versus 29 percent for context editing alone. The full vendor field is messier than those numbers suggest; I compared it in Agent Memory in 2026.

Technique	What it changes in the window	Reported effect	Source
Context editing	Clears stale tool results	84% fewer tokens on a 100-turn search eval; +29% task performance	Anthropic, Sep 2025
Memory tool + context editing	Moves notes out of the window, reloads on demand	+39% on internal agentic search eval	Anthropic, Sep 2025
On-demand tool loading	Defers tool definitions until needed	~77K to 8.7K upfront tokens; Opus 4 accuracy 49% to 74%	Anthropic, Nov 2025
Programmatic tool calling	Keeps intermediate results in code, not context	43,588 to 27,297 average tokens on complex research tasks	Anthropic, Nov 2025
Sub-agents	Isolates exploration in separate windows	+90.2% on internal research eval, at ~15x chat tokens	Anthropic, Jun 2025
Repository context files	Adds static instructions up front	No success gain on average; over 20% more inference cost	Gloaguen et al., 2026

Almost every row is a vendor measuring its own feature, and the one independent row is negative. That is a reason to measure on your own tasks, not to dismiss the techniques.

Sub-agents: when context isolation pays for itself

The isolate verb is the most expensive and the most misunderstood. A sub-agent gets a fresh window, explores with tens of thousands of tokens, and hands the lead agent a condensed result; Anthropic says these summaries are often 1,000 to 2,000 tokens. Claude Code's sub-agent docs frame it the same way: use one when a side task would flood the main conversation with search results or logs you will not reference again.

The cost is real. Anthropic's research system beat a single agent by 90.2 percent on its internal research evaluation, but at roughly fifteen times the tokens of chat, and the same post says most coding tasks have fewer truly parallelisable pieces than research. So isolation pays when the side task is read-heavy and the answer is small: "find every caller of this function", "read these 40 log files and tell me which pod failed first". It does not pay when the sub-agents need to share evolving state. If you are tempted to build a team of agents, read when multi-agent systems help first.

Here is how I express those choices for Claude Code in a project, as .claude/agents/log-triage.md. The sub-agent gets a narrow tool list and a cheaper model, and its whole job is to return a short answer:

---
name: log-triage
description: Reads large log files or kubectl output and returns a short timeline. Use when output would exceed a few hundred lines.
tools: Read, Grep, Glob, Bash
model: haiku
---

You triage logs. Read only what you need, using grep before reading whole files.
Return at most 15 lines: a timestamped timeline of the first failure,
the component that failed, and the exact error string. No raw log dumps.

Why platform discipline still matters for context

None of this is new to anyone who has run production systems. Context is a constrained, shared resource with a cost per unit and non-linear degradation near the limit. We already know how to manage those: budgets, observability, eviction policies, load tests before launch. The teams I see doing context engineering well treat it exactly like that. They log token counts per turn and per tool, they set budgets on tool output, and they run evals when someone adds an MCP server, the same way they would load-test a new dependency. If you are already tracing agent runs, the token breakdown per span is the most useful dashboard you can add; Agent Observability in 2026 covers the plumbing. Context files are configuration, and configuration deserves review, versioning and tests; the AGENTS.md and CLAUDE.md guide walks through that.

FAQ

What is the difference between context engineering and prompt engineering?

Prompt engineering is about wording one instruction well. Context engineering is about everything the model sees on each step: system prompt, tool definitions, retrieved files, message history, tool results and memory. For multi-turn agents most of those tokens come from the loop itself, so curating them matters more than phrasing.

What is context rot?

Context rot is the measured decline in model accuracy as input length grows, even when the answer is present. Chroma's July 2025 study saw it across all 18 models tested, and NoLiMa found 11 of 13 long-context models below half their short-context score at 32K tokens. The fix is fewer, more relevant tokens, not a bigger window.

Do bigger context windows make context engineering unnecessary?

No. A larger window raises the ceiling but does not stop degradation, and every token you keep is billed on every turn. Long windows are useful for one-off analysis of a large document; agents that loop still need compaction, retrieval and tight tool output.

Should every repository have an AGENTS.md or CLAUDE.md?

Probably, but a short one. The 2026 AGENTS.md study found that repository overviews did not help and raised cost by over 20 percent. Keep the file to what an agent cannot infer from the code: exact build and test commands, unusual conventions and hard don'ts.

This quarter I would do three things on any agent I run: log tokens per tool call so I can see where the window goes, cap and summarise every tool's output at the source, and cut the always-loaded context file down to the commands and rules nobody could guess. Then I would measure on real tasks before adding a single sub-agent.

Related: Agent Memory in 2026, RAG vs Long Context vs Fine-Tuning in 2026 and What Are AI Agents? Workflows, ReAct and Real Success Rates (2026).