Evaluating Agents
A model produces one output you can score against a reference. An agent produces a trajectory (a sequence of thoughts, tool calls, and observations) that may reach the right answer by a wasteful or unsafe path, or reach a wrong answer through reasoning that looked fine at every step. Evaluating agents means scoring both what they produced and how they got there.
Real-world examples: measuring whether a coding agent's patch passes the hidden test suite (SWE-bench); checking whether a web agent completed a multi-step task without hallucinating a step (GAIA); auditing whether a support agent followed policy before issuing a refund (τ-bench).
Outcome vs trajectory
Two complementary axes of evaluation:
- Outcome (end-state) evaluation: did the final result satisfy the goal? Often checkable programmatically: tests pass, the order shows refunded, the file contains the value. This is the metric users care about.
- Trajectory (process) evaluation: was the path sound? Did the agent call the right tools, avoid forbidden actions, not loop, and stay within budget? Two agents with identical success rates can have wildly different trajectories — one efficient and auditable, the other a 40-step random walk that got lucky.
A useful efficiency metric normalises steps taken against an oracle minimum:
Pass@k hides flakiness
Agents are stochastic, so a single run is noise. Report pass@k (solved in at least one of k attempts) and pass^k / consistency (solved in all k). A high pass@1 with low consistency means the agent is unreliable, not capable — exactly the distinction a single-run benchmark conceals.
LLM-as-judge on traces
Many trajectory questions have no programmatic checker — "did the agent's reasoning justify the action?", "was the tone appropriate?". Here an LLM-as-judge scores the trace against a rubric. Make it reliable by:
- Using a rubric with explicit criteria rather than a vague "is this good?".
- Preferring pairwise comparison (A vs B) over absolute scores — models are better at ranking than calibrated grading.
- Mitigating bias: judges favour longer answers, their own outputs, and the first option presented. Randomise order and control for length.
Validate the judge before you trust it
An LLM judge is itself a model under evaluation. Spot-check it against human labels on a held-out set and report its agreement (e.g. Cohen's κ) before using it as ground truth. An unvalidated judge can silently bless a regressed agent.
Benchmarks and budgets
Public agent benchmarks pin down capability claims: GAIA (multi-step reasoning + tool use), SWE-bench (resolve real GitHub issues), WebArena (browser tasks), τ-bench (tool-agent-user interaction with policy). They are necessary but not sufficient — your task distribution differs, and benchmarks leak into training data over time.
Two budgets gate every production agent and belong in the eval report:
- Cost: tokens × steps × price. An agent that solves a task for $3 may be a worse product than a workflow that solves 90% of them for $0.02.
- Latency: steps are sequential, so end-to-end time grows with trajectory length. Parallelising independent tool calls is often the highest-leverage fix.
For the full picture — how these metrics organise into four levels (single-agent correctness, per-agent profiling, multi-agent trajectory correctness, and whole-system health), the latency metrics that decide usability, and the leading indicators that predict an effective agent — see the deep dive:
Every piece above is easy to state and hard to size. Build the eval stage by stage below, on two simulated agents with known true skills: success rate with the interval it actually carries, trajectory metrics, pass@k against consistency, a judge scored against human labels, then budgets. The tasks and runs per task controls re-run the whole eval, which is the point, because most of what an eval reports is a function of how many times you looked.
Building an agent eval, step by step
One trajectory is an anecdote
Here is a single run: a sequence of tool calls ending in success or failure. It is the unit everything else is built from, and on its own it tells you almost nothing — an agent is stochastic, so this is one draw from a distribution you have not measured.
It does contain more than a verdict, though. The path is visible: which tools were called, how many steps it took against the 5 an oracle needs, and whether it touched anything it should not have. Outcome and trajectory are both in here; the rest of the walkthrough is about extracting them at a scale where they mean something.
Two agents, two trajectories, same task. Note that the slower one is not obviously the worse one yet.
sample
50 tasks × 3 runs = 150 trajectories per agent.
Two simulated agents with fixed true skills, evaluated the way the lesson describes: outcome, then trajectory, then repeated runs, then a judge that is itself validated, then budgets. Wilson intervals, pass@k and pass^k, Cohen's κ and cost per solved task are all computed from the runs. The task-count and repeat controls re-run the whole eval, which is the point — most of what an eval reports is a function of how many times you looked.
Exercise
Which question is answered by *trajectory* (process) evaluation rather than *outcome* evaluation?
Need a hint?
Outcome = did the final result satisfy the goal. Trajectory = was the path sound?
Exercise
An agent solves a task in 1 of 5 attempts. Why report pass@k AND consistency rather than a single run?
Need a hint?
Agents are stochastic; one run is noise.
Common mistakes
- Scoring only the final answer: missing unsafe or wasteful trajectories that happen to succeed.
- Single-run evaluation: reporting one lucky trajectory instead of pass@k with consistency.
- Trusting an unvalidated judge: using LLM-as-judge as ground truth without checking it against humans.
- Ignoring cost and latency: shipping an agent that is accurate but too slow or expensive to use.
Key takeaways
- Evaluate outcome (did it work) and trajectory (was the path sound, efficient, safe).
- Agents are stochastic — report pass@k and consistency, not a single run.
- LLM-as-judge scores unverifiable trace quality; use rubrics, pairwise comparison, and validate the judge against humans.
- Public benchmarks bound capability; cost and latency budgets decide whether the agent actually ships.
Related concepts
- Reflection Patterns: the self-critique loops that evaluation both measures and powers
- Tool Use, Function Calling & MCP: the trajectory steps being scored
- LLM Evaluation & LLM-as-a-Judge: the judging machinery applied to single-turn outputs
- Observability for Agents: running these outcome and trajectory evals online over sampled production traces