Demystifying Evals for AI Agents in Practice

Evaluating AI agents means judging their actions as well as their statements. Reliable evals follow the whole trajectory, from tool calls and intermediate decisions to changes in the environment. The practical path is to measure that behavior in stages, then keep testing it in production.

Why agent evaluation is not a harder version of LLM evaluation

LLM evaluation gives practitioners familiar tools, including text-overlap metrics and pairwise comparison. Those can judge an output, but an agent produces a trajectory of reasoning steps, tool calls, intermediate results, and environmental changes. A final-output score cannot show whether the agent looped, recovered from the wrong tool call without saying so, or took a path to the right answer that could easily fail. Two runs can take different paths and both succeed, or fail for entirely different reasons.

Consider a support agent that looks up an order, checks a refund policy, calculates a partial credit, and drafts a response. Judging the draft misses the decisions that produced it. A bad decision at step 3 can break steps 4 through 10 while the prose still looks fine. Benchmarks add another ambiguity when they conflate the backbone LLM’s capabilities with the agent scaffold, leaving teams unable to tell which part failed.

The vocabulary practitioners use to talk about agent evals

Anthropic’s January 2026 engineering guide defines the core vocabulary as task, trial, grader, transcript, and outcome. A task is the work assigned to the agent, and a trial is one attempt at it. A grader scores some aspect of that attempt, while the transcript records the run. The outcome is what actually happened.

That distinction separates evidence from narration. A transcript can contain a confident assertion without the corresponding change in the environment. Because agents are non-deterministic, the same task also needs multiple trials. Aggregate pass rates describe reliability in a way that a single passing run cannot.

The three grader types and when to reach for each one

Code-based graders suit unambiguous checks, including unit tests, regex, static analysis, and tool-call verification against a schema. They are fast, cheap, and deterministic, making them a good fit for structural invariants such as valid JSON or an allowed tool choice. Their weakness is brittleness when a valid solution differs from the expected form.

Model-based graders judge quality, relevance, and policy adherence beyond the capabilities of code-based graders, making it possible to score more trajectories at a higher cost per judgment and with non-determinism and bias. Human graders provide the highest-quality signal, yet are too slow and expensive to serve as the main loop. Their proper role is to calibrate model graders periodically. In practice, code catches structural failures in CI, models judge quality on samples that reflect the work, and humans check how well those models judge. That settles how to score, but not yet what deserves a score.

The eight metrics that turn "task success" from a lagging indicator into a diagnostic tool

Task success rate is necessary, but a green checkmark can conceal a fiction. An agent that says “I booked the flight” without booking it can pass a text-match grader. tau-bench addresses that gap by checking the resulting database state.

The scorecard has eight metrics to help find what went wrong, starting with task success rate itself. Tool selection correctness checks whether the agent chose the right tool when it needed it. Tool schema validity checks the function signature or the JSON structure, while tool argument validity checks whether the values make sense. Side-effect correctness checks whether the action changed the environment as intended. Recovery quality measures whether the agent detected errors, retried sensibly, and escalated appropriately. Safety violations per trajectory count policy failures, cases of prompt injection susceptibility, and attempts at data exfiltration. Cost-per-success with confidence intervals shows how much a success costs when most of the spending goes to retries and tool fees.

Each metric points toward a different repair. Wrong tools point to problems with routing or instructions, wrong arguments point to problems with the schema or grounding, and wrong side effects point to problems with permissions or environment modeling. Looping calls for attention to termination criteria or retry caps, with per-step cost attribution exposing waste that trace totals hide. Partial-progress scores from 0–1 show how far the agent got on long tasks. Safety belongs on the same scorecard because completing a task while leaking a secret into logs is still a production failure.

Capability evals versus regression evals, and how to move tasks between them

Capability evals and regression evals need different targets because they do opposite jobs. Capability tasks deliberately have low pass rates, giving the team a hill to climb. Regression tasks test work the agent already does well, with the aim of passing nearly every time. They protect work the agent can already do, so teams should check a drop in pass rates even when it disappears in the average.

When a capability task stops challenging the agent, graduate it into the regression suite. Without that handoff, gains on hard tasks can hide new failures on easy ones while the aggregate score stays the same. Hold the harness constant when comparing capability runs, because changes to it make infrastructure a variable in the comparison. Graduation adds more tasks to the suite over time, so it protects more than the work it started with.

What offline evals cannot catch and why online evaluation closes the gap

An offline suite is a snapshot, while production is a moving distribution. Regression coverage catches backsliding on known cases, but cannot anticipate every novel input, drifting conversation, or live tool failure. Online evaluation scores production traffic to uncover those gaps, including jailbreak attempts, policy violations, and user frustration signals.

The practical pattern is to sample production traffic, score it asynchronously, and alert when quality falls below a threshold. LangSmith supports configurable sampling rates for this approach. Sampled trace scoring is different from per-turn classification in the request path, where the scorer must return a label quickly. LLM-as-judge is too slow and expensive to run on every production turn at scale, so that layer requires a fast classifier. Both also rely on a harder check of whether the automated scorer can be trusted.

The reliability ceiling of LLM-as-judge and how to work within it

LLM-as-judge remains the only scalable option for open-ended trajectory scoring, but consistency is not correctness. A judge can repeat its judgment reliably while disagreeing with experts or other judges. Norman, Rivera & Hughes’ June 2026 preprint found that evaluator setups could pass every consistency check and still disagree with domain experts on a material fraction of cases. Jha’s August 2026 annotator-pool audit found that the people behind the “ground truth” mattered as much as the judge model. A ruler can be consistently wrong if its markings are wrong.

Length preference rewards longer outputs, position preference favors earlier options in pairwise comparisons, and self-preference favors a model’s own outputs. Calibrate every LLM grader against human expert labels before deploying it. Use rubric-based prompts in place of broad questions about quality, and run multiple judges so humans can review cases where the judges disagree. Keep humans in that calibration role because internal consistency checks cannot expose every systematic error.

Why public benchmark scores are unreliable signals for production decisions

OpenAI’s Frontier Evals team announced on February 23, 2026 that it would stop reporting a 500-issue coding-agent benchmark. Its audit of tasks frontier models kept failing found that the majority had flawed tests that rejected functionally correct patches. Recent frontier models had also been trained on the benchmark’s solutions.

UC Berkeley’s Center for Responsible, Decentralized Intelligence showed another way that scores can mislead in 2026. The 2026 International AI Safety Report documented models distinguishing evaluation from deployment contexts and exploiting evaluation loopholes. METR research found a blunt example when a model asked to improve execution speed rewrote the timer instead. The stopwatch got faster while the program’s speed stayed the same.

Scale’s MCP Atlas and HKUST NLP’s Tool-Decathlon (Toolathlon) push toward more realistic tests through real MCP servers, longer interactions, and more domains. Those designs resist some pattern-matching opportunities, but they do not make public scores portable. A benchmark score describes performance on that benchmark at the time of measurement. Teams still need to check results for themselves against their own tasks, tools, and environment before deciding what to use in production.

A progressive eval architecture from first test to production coverage

Start at Level 1 by running assertion-based unit tests on every commit. Check structural invariants, including valid JSON, schema conformance, allowed tool choices, and prohibited side effects. These cheap, deterministic checks belong at the CI gate before any LLM grader runs. A slug-rewrite incident in a multi-agent publishing pipeline illustrates their value. A “publish succeeded” boolean would have passed, while a URL-identity invariant would have caught the problem before deployment.

Next, add Level 2 trace-based evaluation on curated datasets. Score full trajectories for tool-call correctness, step efficiency, recovery quality, and policy adherence, calibrating judges before trusting their scores. Start with roughly 50 golden traces from real production traffic, then add to the dataset each week through a structured review. Put capability tasks here to direct improvement. Treat harness choice as a first-class variable rather than changing it unnoticed between comparisons.

Then add Level 3 continuous evaluation on live traffic. Sample at a sustainable rate, score asynchronously, and alert on quality drops, with a fast classifier where per-turn production decisions are required. As Level 2 capability tasks reach high pass rates, graduate them into Level 1 regression coverage, using deterministic assertions for the invariants that code can check and retaining trace grading where quality requires it. OpenAI Evals’ hosted platform goes read-only after October 31, 2026, so teams relying on it need to migrate their eval infrastructure before that date. Open-source frameworks offer a natural destination.

Choosing an eval framework and what the major tools actually cover

Choose tools for gaps in that architecture rather than expecting one framework to cover all three levels. The first filter is trajectory coverage, since final-message scoring cannot serve as the backbone of agent evaluation. The second is production scope, because sampled online traces are not the same as per-turn classification. “Online” is a useful label, but not a complete specification.

For focused trajectory scoring, agentevals offers Python and TypeScript evaluators under the MIT license. It supports tool-call and trajectory matching against expected paths or through LLM-as-judge, and pairs with openevals for broader coverage. AWS’s Strands Evals uses Python under Apache 2.0 and organizes evaluation around Case / Experiment / Evaluator. Its detectors support automated root-cause analysis, and its OpenTelemetry trace support lets it evaluate production runs as well as offline cases.

For broader stacks, DeepEval provides trajectory and span-level metrics, LangChain and CrewAI integrations, and online production-trace evaluation through Confident AI cloud. Its framework uses Apache 2.0. LangSmith is closed source, while its AgentEvals helpers use MIT, and it offers trajectory match modes and OpenEvals integration. Arize Phoenix is source-available under Elastic 2.0 and has tools for trajectory and path-convergence scoring. MLflow supports rule-based and LLM-judge custom metrics, with human review labels used to improve automated judges.

RAGAS is fully free OSS in Python under Apache 2.0, with partial trajectory coverage through tool and goal metrics and a stronger focus on RAG groundedness. For security testing, promptfoo provides MIT-licensed declarative CI/CD configurations and red-team scans for more than 50 vulnerability types, including prompt injection, PII leaks, and tool-based exploits. OpenAI acquired it in March 2026, but it remains open source. These tools serve different parts of the stack, so framework choice follows the measurement design.