// agent_quality
Evaluating AI agents
An agent that answers correctly in the demo and deletes the wrong record on the fortieth run has not been evaluated — it has been watched. Evaluating agents means checking what they did to the world, how they got there, and whether they do it every time.
The short version
Grade the end state with code wherever you can, grade the trajectory for what code can't see, and run every task several times — an agent that succeeds 90% of the time fails more than half of eight-run streaks.
Run it in a sandboxed environment with realistic state, track cost and steps per task alongside success, and turn every production failure into a new test.
What to measure
did it work
Outcome
how
Trajectory
every time
Reliability
at what cost
Efficiency
safely
Safety
for the user
Communication
Reliability: pass@k versus pass^k
pass@k asks whether an agent succeeds at least once in k attempts — a capability measure, appropriate when a verifier can pick the good attempt. pass^k asks whether it succeeds in all k attempts — a reliability measure, appropriate when the agent acts for a user and gets one shot. For agents in production, pass^k is usually the honest number.
| Per-attempt success | pass^1 | pass^4 | pass^8 | pass^16 |
|---|---|---|---|---|
| 99% | 99% | 96% | 92% | 85% |
| 95% | 95% | 81% | 66% | 44% |
| 90% | 90% | 66% | 43% | 19% |
| 80% | 80% | 41% | 17% | 3% |
Real attempts are not fully independent — some tasks are simply hard — so measure pass^k directly by rerunning tasks rather than inferring it from a single-run success rate.
Building the evaluation
- 01
Collect tasks from real traffic
Sample real requests and past failures, not invented ones. Cover each task type, the ambiguous cases, and requests the agent should refuse or escalate. - 02
Build a sandboxed environment
Give the agent the same tools against a resettable copy of realistic state — a seeded database, a mock API that behaves like the real one, a disposable repository. Reset between runs so results are comparable. - 03
Write end-state checks in code
For each task, assert on the resulting state rather than on the agent’s final message. Where outcomes can’t be checked in code, write a rubric for an LLM judge and calibrate it against human grades. - 04
Run each task k times
Report success, pass^k, steps, tokens and cost per task. Look at the distribution, not just the mean — a long tail of 40-step runs is a production incident waiting to happen. - 05
Gate changes on it
Rerun on every prompt, tool, model or retrieval change and block regressions. A model upgrade that improves average success but lowers pass^k is not an upgrade for an agent that acts on users’ behalf. - 06
Close the loop from production
Trace every step in production — model calls, tool calls, arguments, results — using a standard such as the OpenTelemetry generative-AI conventions, and turn each failure into a new eval task.
Don’t grade the final message
Public benchmarks: a shortlist, not a verdict
LiveBench's agentic coding category is one of the better public signals for multi-step tool use. Today's top five, live:
| # | Model | Agentic coding | Blended / 1M |
|---|---|---|---|
| 1 | DeepSeek V4.1 Flash | 77.3 | $0.200 |
| 2 | Claude Opus 5.5 | 71.7 | $8.00 |
| 3 | Claude Fable 5.1 | 66.1 | $20.00 |
| 4 | Claude Opus 5 | 65.2 | $10.00 |
| 5 | DeepSeek V4 Flash Vision Exp | 65.1 | $0.330 |
Use rankings like this — or the full agentic coding ranking — to pick two or three candidates, then decide with your own task set. Scores measure someone else's tools and environment; yours will rank models differently.
Common mistakes
Single-run results
Mocks that are too kind
Ignoring cost per task
A frozen task set
Agent evaluation FAQ
How is evaluating an agent different from evaluating an LLM?
An LLM call is judged on one output. An agent takes many steps, calls tools, changes state in the world and can reach a correct result by several paths — or a wrong one that looks fine in the final message. Agent evaluation checks the end state the agent produced, how it got there, and how consistently it succeeds across repeated runs.
What is pass^k?
pass^k is the probability that an agent succeeds on a task in all k independent attempts. It measures reliability, which matters when an agent acts for a user and must work every time. It falls quickly: a task solved 90% of the time has a pass^8 of about 43% if attempts are independent. pass@k — at least one success in k attempts — measures capability instead.
Should I use LLM-as-judge for agents?
Use code to check outcomes whenever the end state is checkable — the database row exists, the tests pass, the refund was issued once. Use an LLM judge for what code cannot check, such as whether the agent’s messages were accurate and polite or whether its path was reasonable, and calibrate the judge against human labels first.
Are public agent benchmarks useful?
They are useful for shortlisting models, not for deciding. A benchmark measures someone else’s tasks, tools and environment. Your agent’s success depends on your tools, prompts and data, so the deciding evaluation is a task set drawn from your own traffic.
How many test tasks does an agent eval need?
Enough to cover your main task types and known failure modes, run several times each. A few dozen well-chosen tasks with checkable end states and repeated trials tell you more than hundreds of single-run tasks graded by eye. Grow the set with every production failure.