Evals for AI agents: start with twenty tasks, grade the outcome
How to test an AI agent before users do: tasks, trials, graders, pass@k versus pass^k, and a first eval set you can build in an afternoon.
You change a prompt, and the agent seems better in the three cases you tried. Two days later, a user finds a case you broke. Most teams building agents hit this within weeks, and the fix is the same as in any other software: automated tests. For agents we call them evals.
Anthropic's engineering team published a thorough guide on this, Demystifying evals for AI agents, in January 2026. This article is my own working summary of its main ideas, with a practical plan for getting started.
Why agents are harder to test
A single-turn model call is easy to test: one input, one output, compare. An agent acts over many turns, calls tools, changes the state of the world, and may find a route to the goal that you did not imagine. The guide says exactly this: autonomy and flexibility are what make agents useful and also what make them hard to evaluate.
If you are still deciding whether you need an agent at all, see Workflow or agent?. Fixed workflows are easier to test, which is one more reason to prefer them when they are enough.
The vocabulary
The guide uses a small set of terms, and having them straight makes the rest easier:
- Task: one test, with an input and clear success criteria.
- Trial: one attempt at a task. You run several, because results vary.
- Grader: logic that scores some part of what happened. A task can have several.
- Transcript: the full record of a trial, including tool calls and intermediate results.
- Outcome: the final state of the environment. A booking exists in the database, or it does not.
- Harness: the infrastructure that runs tasks, records them and aggregates scores.
The distinction between transcript and outcome is the important one. An agent can say "I booked your table" without a booking existing. Grade the state of the world, not the agent's claim.
Three kinds of graders
| Grader | Good at | Weak at |
|---|---|---|
| Code-based (string checks, tests, state checks, tool-call checks) | Fast, cheap, objective, repeatable | Brittle, misses nuance |
| Model-based (rubrics, pairwise comparison) | Open-ended output, tone, quality | Non-deterministic, costs more, needs calibration |
| Human (expert review, spot checks) | Gold-standard quality | Slow and expensive |
The guide's advice on order is simple: prefer deterministic graders where you can, use a model grader where you must, and use humans to check the model grader. Calibrate a model judge against human opinions before you trust its scores, and consider allowing it to answer "unknown" instead of forcing a verdict.
Why one number is not enough: pass@k and pass^k
Agents are not deterministic, so you run each task more than once. Two ways of summarising the trials answer different questions:
- pass@k: the chance of at least one success in k tries. It rises with k. It fits cases where one good answer is enough.
- pass^k: the chance that all k tries succeed. It falls with k. It fits cases where users need consistency.
The guide gives a worked example: with a 75% success rate per trial, passing three of three is about 42% (0.75 cubed). An agent that looks fine on a single run can look quite different when you ask for reliability.
p = 0.75
k = 3
pass_all_k = p ** k # 0.421875
pass_at_least_one = 1 - (1 - p) ** k # 0.984375
For anything customer-facing, report pass^k as well.
A plan you can finish in an afternoon
The guide's roadmap runs from step 0 to step 8. Here is the part that gets you started.
- Start early and small. Twenty to fifty tasks drawn from real failures are enough, according to the guide. Waiting makes it harder.
- Turn manual checks into tasks. Your pre-release checklist, bug tracker and support tickets are a ready supply.
- Write tasks that two experts would grade the same way. If a task fails on every trial, suspect the task before the agent.
- Test both directions. Include cases where the behaviour should happen and cases where it should not. Otherwise you tune an agent to do too much.
- Start every trial clean. Shared state between trials causes correlated failures or unfair advantages.
- Grade the outcome, not the exact path. Agents find valid routes you did not plan for. Give partial credit for multi-step tasks.
- Read the transcripts. This is how you tell an agent mistake from a grader bug.
Note
The guide describes a model scoring 42% on a benchmark because of grading bugs, for example penalising an answer of "96.12" when the reference was a longer number. After the fixes, the score was 95%. A low score can be a broken test.
Capability evals and regression evals
Two kinds of suites do different jobs. A capability eval starts with a low pass rate and gives you a hill to climb. A regression eval should sit near 100% and tells you when something broke. When a capability eval is mostly solved, it can graduate into the regression suite.
Watch for saturation. At 100%, an eval can only catch regressions, and it hides real improvements, so you need to add harder tasks over time.
Evals are one layer
The guide compares testing to the Swiss cheese model: each layer has holes, and layering catches what any one misses. Automated evals sit beside production monitoring, user feedback, A/B tests and manual transcript review.
Evals also pair well with limiting what the agent can do in the first place. A narrow tool set, as in Giving an AI agent only the access it needs, makes the space of outcomes smaller and the grading simpler.
A minimal eval record
One way to keep tasks simple is a small, boring file format:
{
"id": "refund-over-limit",
"input": "Customer asks for a refund of 900 on an order of 1000.",
"setup": "fresh test database, order 1000 exists",
"checks": [
{"type": "state", "assert": "refund row absent"},
{"type": "state", "assert": "approval request created"},
{"type": "rubric", "assert": "reply is polite and explains the approval step"}
],
"trials": 5
}
Two state checks and one rubric. The state checks catch the costly mistakes; the rubric covers tone.
Where to begin tomorrow
Open your bug tracker, pick the ten most embarrassing agent failures, and write each as a task with a state check. Run each five times. You will have a baseline by lunch, and the next prompt change will have something to answer to.
