Anthropic Effective Context Engineering for AI Agents

TL;DR

Anthropic announced context engineering, a set of techniques for selecting and managing the tokens fed to a large‑language model during inference, and showed how it improves the reliability and scalability of AI agents.

What is Context Engineering?

Context engineering is the practice of curating the exact set of tokens (system prompts, tool definitions, examples, message history, etc.) that a language model sees at each inference step. It extends prompt engineering by treating the model’s limited attention budget as a scarce resource that must be allocated efficiently.

Key points:

  • Context = all tokens present when the model samples a response.
  • Engineering = optimizing those tokens to maximize the probability of the desired behavior while respecting the model’s context‑window limits.
  • The process is iterative; the token set is re‑evaluated before every model call.

Why Context Engineering Matters

LLMs suffer from context rot: as the number of tokens grows, recall accuracy degrades. This mirrors human working‑memory limits and stems from the transformer architecture’s O(n²) attention cost and training bias toward shorter sequences.

Consequences:

  • Every additional token consumes part of the model’s “attention budget”.
  • Longer contexts lead to reduced precision in information retrieval and long‑range reasoning.
  • Even models with extended windows exhibit a gradual performance gradient rather than a hard cutoff.

Therefore, treating context as a finite resource and minimizing low‑signal tokens is essential for building capable agents.

Core Components of Effective Context

System Prompts

  • Must be clear, concise, and written at the right altitude: specific enough to guide behavior but flexible enough to avoid brittle logic.
  • Recommended structure: separate sections (e.g., <background_information>, <instructions>, ## Tool guidance, ## Output description) using XML tags or markdown headers.
  • Start with a minimal prompt; add instructions or examples only after identifying failure modes.

Tools

  • Tools act as contracts between the agent and its environment; they should be token‑efficient and unambiguous.
  • Design tools to be self‑contained, robust, and with descriptive input parameters.
  • Avoid bloated tool sets; a minimal viable tool suite reduces decision ambiguity and context overhead.

Examples (Few‑Shot Prompting)

  • Provide a diverse, canonical set of examples rather than an exhaustive list of edge cases.
  • Examples serve as “pictures worth a thousand words” for the LLM, conveying expected behavior efficiently.

Message History & Retrieval

  • Curate history to retain high‑signal exchanges while discarding redundant tool calls and results.
  • Use just‑in‑time retrieval: keep lightweight identifiers (file paths, URLs) and load data via tools only when needed.

Dynamic Context Retrieval & Agentic Search

Anthropic’s agents now combine pre‑inference embedding retrieval with runtime “just‑in‑time” loading. Agents keep references (e.g., file paths) and fetch content on demand using tools, mirroring human cognition.

Benefits:

  • Storage efficiency – large datasets are never fully loaded into the context window.
  • Metadata signals – filenames, timestamps, and hierarchy guide relevance judgments.
  • Progressive disclosure – agents incrementally discover and incorporate context, maintaining a focused working memory.

Trade‑offs:

  • Runtime retrieval adds latency.
  • Requires careful tool design to prevent wasted context on dead‑ends.

Hybrid strategies (pre‑loaded essential data + just‑in‑time exploration) work well for relatively static domains such as legal or finance.

Techniques for Long‑Horizon Tasks

When tasks exceed the model’s context window, Anthropic recommends three complementary strategies.

1. Compaction

  • Summarize near‑limit conversation histories into a high‑fidelity summary.
  • Re‑initialize the context window with the summary plus the most recent critical tokens.
  • Example: Claude Code clears old tool results and injects a concise summary, preserving architectural decisions while discarding noise.

2. Structured Note‑Taking (Agentic Memory)

  • Agents write persistent notes (e.g., NOTES.md) outside the context window.
  • Notes are re‑imported when needed, providing a lightweight memory layer.
  • Demonstrated in Claude playing Pokémon, where the agent tracks thousands of steps via external notes.
  • Anthropic released a memory tool on the Claude Developer Platform (public beta) to simplify this workflow.

3. Sub‑Agent Architectures

  • Decompose a large problem into specialized sub‑agents, each with its own focused context.
  • Sub‑agents perform deep exploration and return distilled summaries (1‑2 k tokens) to a coordinating lead agent.
  • This pattern yielded substantial gains on complex research tasks (see Anthropic’s multi‑agent research system).

Choosing a technique depends on the task:

  • Compaction for conversational back‑and‑forth.
  • Note‑taking for iterative development with clear milestones.
  • Sub‑agents for parallel, research‑intensive workloads.

Practical Guidance for Engineers

  1. Treat context as a budget – aim for the smallest high‑signal token set.
  2. Iterate – after each failure, refine prompts, tool definitions, or retrieval logic before adding more tokens.
  3. Start minimal – test with the best available model using a bare‑bones prompt, then augment as needed.
  4. Curate tools – keep the toolset lean and ensure each tool’s output is token‑efficient.
  5. Leverage memory tools – store summaries, notes, and metadata outside the context window to avoid pollution.
  6. Hybrid retrieval – combine pre‑computed embeddings with just‑in‑time loading for balanced speed and relevance.

Implications for the Future of AI Agents

  • Shift from prompt‑centric to context‑centric engineering as models become more capable.
  • Agent autonomy will increase; smarter models will need less prescriptive prompting but will still rely on efficient context curation.
  • Tool and memory ecosystems will become core infrastructure for scalable agents.
  • The principle of “find the smallest set of high‑signal tokens” will remain a cornerstone, even as context windows grow.

Getting Started

Anthropic invites developers to experiment with context engineering on the Claude Developer Platform. Resources include:

By applying the techniques above, teams can build agents that remain focused, reliable, and scalable across long‑horizon tasks.

Sources