How to Evaluate AI Agents: Metrics, Tests, and the Approval Gate

Evaluating an AI agent means scoring a multi-step system, not a single prompt and response. A prompt eval asks one question: was this output good? An agent eval has to answer at least four: did the agent achieve the final outcome (system evaluation), was each step in the trajectory correct (process evaluation), did it select the right tools with the right arguments, and does it behave consistently when you run the same task repeatedly. Microsoft's agent evaluator framework makes this split explicit, separating system evaluation (task completion, task adherence, intent resolution) from process evaluation (tool selection, tool input accuracy, tool output utilization, tool call success). Arize frames the same idea as four scopes: outcome assessment, trajectory analysis, decision evaluation, and reliability testing. If your eval only looks at final answers, you will miss agents that reach the right outcome through policy violations, redundant tool calls, or silent mid-chain errors that happen to cancel out.

Measure agent performance with a small set of metrics that cover outcome, process, and cost. The core set: task success rate (the percentage of tasks completed to spec, judged against explicit success criteria), tool selection accuracy (did it pick the right tool and skip unnecessary ones), tool-call or argument correctness (were the parameters grounded, typed, and formatted correctly), step or trajectory accuracy (did the run include the required steps in an acceptable order), groundedness (are claims in the output supported by retrieved context or tool results rather than hallucinated), guardrail and policy violations (a release-blocking count, not an average), and operational metrics: cost per task and latency at p95, not just the mean. Azure's tool input accuracy evaluator is a useful template for argument checking; it scores groundedness, type compliance, format compliance, required parameters, no unexpected parameters, and value appropriateness. Treat success rate as the headline, violations as the guardrail, and the process metrics as diagnostics that tell you where a failing run actually broke.

Test an agent offline first, against a curated dataset with known expected outcomes, before it ever touches live traffic. LangSmith's evaluation guidance distinguishes offline evaluation (pre-deployment runs on curated datasets with reference outputs, where correctness can be verified) from online evaluation (monitoring live production traffic without reference answers). Start small: a manually curated set of 5 to 10 examples per component that define your quality bar, drawn from real tasks, then grow the set every time you find a new failure mode. Each example needs the input, the expected outcome, and ideally the expected tool sequence. Run the full suite on every prompt, model, or tool change, exactly like a regression test suite. Arize describes the production workflow as a loop: define expected behavior, build datasets, trace runs, choose evaluators, investigate failures, and add every new failure back into the suite as a regression case. The dataset is the asset; the eval harness is just plumbing.

Evaluate the trajectory by comparing the sequence of steps the agent took against an expected sequence, and evaluate tool calls by checking selection and arguments deterministically wherever possible. Trajectory matching comes in strictness tiers: exact match (order and content must match ground truth), in-order match (all required steps present in order, extra steps allowed), and any-order match (all required steps present, order free). Azure's task navigation efficiency evaluator implements exactly these three modes and reports precision, recall, and F1 against the expected action list. For tool calls, prefer code checks over judges: tool name equality, argument schema validation, and value assertions are deterministic and cheap. Confident AI's guide breaks tool evaluation into tool correctness, argument correctness, and step efficiency (flagging unnecessary calls and loops), and stresses component-level evaluation because errors compound across steps; an agent can look successful end to end while making expensive or silently wrong moves deep in the chain. Tracing matters here: every failed metric should link to the specific span that failed.

Unit test the deterministic parts with code, and stress test the whole system with adversarial and repeated runs. Unit tests cover tool wrappers, argument validation, retry logic, and output schemas; these are ordinary software tests and should be fast and exact. For the agent loop itself, use three grader types with known tradeoffs: code-based graders (fast, objective, brittle to valid variation), model-based graders (flexible, handle nuance, but non-deterministic and needing calibration), and human graders (gold standard, slow, expensive). Stress testing means red-teaming with malformed inputs, ambiguous requests, prompt-injection attempts inside tool results, and tool failures, plus reliability testing across repeated runs. Anthropic's engineering guidance introduces two metrics for non-determinism: pass@k, the probability the agent succeeds at least once in k attempts, and pass^k, the probability all k attempts succeed. They diverge sharply; at k equals 10, pass@k can approach 100 percent while pass^k collapses toward zero. For agents that act on production systems, pass^k is the honest number, because a user does not get k attempts at a CRM write.

Use an LLM judge for criteria that need semantic interpretation (was this summary faithful, was the explanation adequate, did the reply match the requested tone) and never for things a code check can verify. The known pitfalls: judges are non-deterministic, they drift from human preferences without calibration, and they can hallucinate justifications for wrong scores. The mitigations are consistent across vendors: give the judge an explicit out (an Unknown or abstain option) so it does not force a verdict, use structured rubrics that score one isolated dimension at a time rather than a single holistic grade, and calibrate regularly against human-labeled examples. Arize and Anthropic both emphasize that human review of transcripts remains essential to confirm the judge is measuring what actually matters, and Confident AI recommends binary or threshold outputs over raw scalar scores because pass/fail decisions are easier to calibrate and act on. Treat the judge as a component with its own eval, not as ground truth.

In production you switch from reference-based scoring to monitoring, sampling, and controlled exposure. Online evaluation runs on live traffic without reference answers, watching quality patterns and anomalies: success and escalation rates, guardrail violations, tool error rates, retries, p95 latency, and cost per task. Before full exposure, use shadow mode (the agent runs on real inputs but its actions are not executed, so you can score its proposals against what a human actually did) and A/B or pairwise comparison between versions to confirm a change is an improvement rather than a regression. Keep human annotation in the loop by sampling production traces for review, prioritizing low-confidence runs and high-consequence actions. This layered approach maps cleanly onto the NIST AI Risk Management Framework's Measure and Manage functions: quantify performance and risk continuously, then apply controls proportionate to what the measurements show. The operational rule is simple: any action class whose measured reliability you cannot yet defend stays gated or shadowed.

Voice agents add a speech layer on top of everything above, so you evaluate the stack: transcription, turn-taking, latency, and call outcomes. Transcription quality is measured by word error rate, computed as substitutions plus deletions plus insertions divided by total words spoken; under 5 percent is the enterprise bar, 5 to 10 percent is workable, and Deepgram notes that off-the-shelf recognition on real call audio can run near 25 percent, with the caveat that which words are wrong matters more than the raw rate. Turn-taking is scored by end-of-turn detection and interruption recovery, with a common target above 90 percent successful barge-in recovery. Latency targets are aggressive because humans expect roughly 300 millisecond conversational gaps; Hamming's benchmark guide targets p95 turn latency under 800 milliseconds while reporting production medians of 1.4 to 1.7 seconds. The business metrics are containment rate (calls resolved with no human escalation, where 70 percent or more is considered good and mature deployments target 75 to 85 percent) and task success rate, which varies by use case, with appointment scheduling targeting above 90 percent. Escalation is not failure; a voice agent that hands off cleanly at the right moment scores better than one that contains a call and completes the wrong task.

You gate the actions you cannot yet trust the numbers on. This is the step most eval writeups skip: evaluation and human approval are complements, not alternatives. For agents that only draft text, an occasional failure costs an edit. For agents that write to a CRM, file a ticket, or send an email to a customer, a failure is a wrong record, a broken SLA clock, or a message you cannot unsend, and pass^k style reliability is the standard that matters. The pragmatic pattern is a review gate in front of every write: the agent proposes the full action with its evidence, a human approves, edits, or rejects, and the approval stream becomes your best evaluation dataset, because every edit is a labeled failure and every rejection is a red-team example you did not have to invent. As measured reliability rises for a given action class, you can widen autonomy for that class specifically, not for the agent as a whole. This is how we run post-call execution at Mindlyft: after a customer call, ASTRA drafts the CRM updates, tickets, and follow-up emails the conversation committed to, and every write waits in a review console for a CSM, SE, or TAM to approve before it ships, so nothing promised on a call gets dropped and nothing unreviewed gets written. If you want to see it on your own post-call workflow, the first workflow is free, then $5,995 per month at mindlyft.in.