vvek

Context engineering for agents: give the model less, but better

A longer context is not a better one. Practical ways to keep an agent's context small and useful: just-in-time retrieval, compaction, notes and sub-agents.

When an agent starts to behave badly on a long task, repeating itself, forgetting a decision from an hour ago, or picking the wrong tool, the first thing people do is rewrite the prompt. Often the prompt is fine. The problem is everything else in the context window.

Anthropic's engineering team calls the discipline of managing that context engineering in their write-up on the subject. This article is my own working summary, with some practical additions.

Prompt engineering versus context engineering

Prompt engineering is about writing good instructions. Context engineering is wider: it is the curation of everything the model sees on each call. That covers the system prompt, tool definitions, message history, files, and search results added during the run.

For a single question and answer, the difference is small. For an agent that runs for dozens of steps, it is large, because at every step something decides what goes into the next call. The article frames it as finding the context setup most likely to get the behavior you want.

Why more is not better

Models don't use their context evenly. The article describes context rot: as the window fills with tokens, the model's ability to recall details from it gets worse. Models differ in how gently they degrade, but the trend is general.

Think of it like an attention budget. Every token you add spends some of it. The guiding sentence from the article is worth keeping near your desk: aim to "find the smallest set of high-signal tokens that maximize the likelihood of your desired outcome."

Notice what this is not. It is not a call for short prompts at all costs. A missing instruction is as bad as a noisy one. It is a call to make each token earn its place.

Four techniques that work

1. Retrieve just in time

Instead of loading every possibly relevant document at the start, keep lightweight references, such as file paths, saved queries or links, and let the agent fetch the content when it needs it, using tools.

This mirrors how a person works. You do not memorise a whole codebase; you know where things are and look them up. The article notes that a hybrid is common: preload a small amount that is nearly always needed, and let the agent explore for the rest.

The cost is that exploration uses tool calls, and a vague tool description can send an agent off track. Clear tool names and descriptions matter here; see Giving an AI agent only the access it needs for why a small, well-defined tool set also helps for safety.

2. Trim tool results

Tool output is the fastest way to flood a context. A query that returns two thousand rows, or a page fetch that returns the whole page, can push out everything useful.

Shape results before they reach the model:

def search_orders(query: str, limit: int = 10) -> dict:
    rows = db.search(query, limit=limit)
    return {
        "count": len(rows),
        "truncated": len(rows) == limit,
        "orders": [
            {"id": r.id, "status": r.status, "total": r.total}
            for r in rows
        ],
    }

Return the fields the agent needs, cap the count, and say plainly when the result was cut off, so the agent can narrow its query instead of assuming it saw everything. The article also describes clearing old tool results from the history once they are no longer needed, a light form of compaction.

3. Compact long histories

When a conversation nears the limit, you can summarise it and start a new window from the summary. The hard part is deciding what to keep. Decisions made, constraints discovered, open problems and the current plan usually survive; raw outputs and dead ends usually do not.

Compaction loses information by design, so test it. Take a long transcript, compact it, and check whether the agent still behaves correctly from the summary. The habits in Evals for AI agents apply directly: a few long-running tasks with state checks will tell you quickly whether your compaction drops something important.

4. Write notes outside the window

Let the agent keep a notes file or a to-do list in persistent storage, and read it back when it needs it. The article calls this structured note-taking, or agentic memory. A progress file that says what is done, what is next and what was decided lets an agent pick up after a compaction, or after a restart, without guessing.

Keep the notes short and structured. A notes file that grows without limit becomes a context problem of its own.

Sub-agents as context boundaries

A fifth idea ties the others together: split the work. A lead agent hands focused sub-tasks to sub-agents, each starting with a clean window. A sub-agent may read a lot, but it returns only a condensed summary, which the article puts at roughly one to two thousand tokens, so the lead agent's context stays small.

This is a context-management tool before it is a speed tool. It also adds cost and complexity, so use it when a task really does involve large amounts of exploration. If you are not sure you need multiple agents at all, Workflow or agent? is a good place to start.

A short checklist

  • Can you list what is in your agent's context at step 20? If not, log it.
  • Which parts of it are stable, and which change each step? Put the stable parts first; this also helps if your provider supports prompt caching.
  • Are tool results capped and shaped?
  • What happens when the history gets long: truncation, summary, or failure?
  • Does the agent have a place to write down decisions?
  • Do you have a task that runs long enough to expose all of the above?

Tip

Print the full context of a real failing run and read it top to bottom, as the model would. Most context problems are obvious within a minute: duplicated instructions, a giant tool result, a stale plan.

Where to start

Pick one long-running task your agent does badly. Log the context size at each step. Find the biggest single contributor, usually a tool result, and trim it. Run the task again. That one change often does more than a week of prompt rewording.

Sources

Keep reading

A short email when we publish. Confirm once, unsubscribe any time.