How do you evaluate an AI agent?

Score the whole run and not only the final answer. Measure task completion, correctness of tool calls, grounding of facts, policy violations, escalation behaviour, cost per run and latency, using a test set built from real work. Run it on every change to the prompt, model or tools.

What is the difference between evaluating an LLM and an AI agent?

An LLM evaluation mostly checks an output. An agent evaluation also checks the path: which tools were called, with what arguments, in what order, and whether the agent stopped or escalated at the right moment.

How many test cases does an AI agent evaluation need?

Start small, with 30 to 50 real cases that include known failures and difficult edge cases, then add every production failure you find. Coverage of the failures that matter counts for more than the size of the set.

November 5, 20263 min read

AI Agent Evaluation: A 7-Metric Scorecard That Goes Beyond Accuracy

An agent can give the right final answer and still be unsafe, slow or ruinously expensive. Here is a seven-metric scorecard for AI agent evaluation, how to build a test set from real work, and how to set release gates before you ship.

By Vikas Goel

"It's 92% accurate" tells you very little about an agent.

An agent can reach the right answer after calling the wrong tool, spending ten times the budget, or skipping a step that a regulator cares about. A final-answer score would mark that run as a pass.

Seven-metric AI agent scorecard with policy violations as a hard gate at zero

The scorecard

Score the whole run. These are the seven metrics I'd start with:

Metric	The question it answers	Type
Task completion	Did it finish the real job?	Target
Tool-call correctness	Right tool, right arguments, right order?	Target
Grounding	Can each fact be traced to a source?	Target
Policy violations	Did it do anything it must never do?	Hard gate: zero
Escalation quality	Did it hand over when it should, and only then?	Target
Cost per run	Is it still affordable at volume?	Limit
95th-percentile latency	Are the slow runs acceptable?	Limit

The important column is the last one. A hard gate can't be traded away. Better completion doesn't excuse a policy violation, so don't merge everything into a single score that lets one hide the other.

Build the test set from real work

Start with 30 to 50 real cases. Use anonymised production data, or real work from the team doing the job today.
Include the difficult ones. Ambiguous requests, missing information, and cases where the right behaviour is to refuse.
Write down what good looks like. Include expected tool calls and forbidden actions, as well as the answer.
Add every production failure. A failure you turned into a test case can't quietly come back.

Run it like software testing

Agents are probabilistic, so a single pass proves little. Run each case several times and look at the pass rate and the spread.

Run the suite when the prompt, model, tools or retrieved data change. Model swaps deserve particular suspicion: a new version can improve your overall score and still break one workflow you depend on.

Use code to check what code can check, such as schemas, tool arguments, cost and forbidden actions. Keep model-based judges for the subjective parts, and test those judges against human labels, because a judge can be wrong too.

Set the release gate before you see the results

Decide your thresholds first. If you pick them after the run, you'll tend to approve whatever you got. A simple gate looks like this:

Zero policy violations
Task completion at or above the level of the current process
Cost per run within budget (the AI agent cost calculator helps set this)
No regression on any previously fixed failure

This is how you show an agent is ready for production rather than simply hoping. It's also the measurement side of deterministic AI agents.

Vikas Goel is the founder of Thinkerwave AITech and a former enterprise CTO. He works with enterprises and founders as a fractional AI CTO and AI advisor.

AI Agent Evaluation
AI Agents
Production AI
AI Agent Reliability
LLM Evaluation

← Previous

AI Agent Security: Assume the Prompt Will Be Injected, Then Limit the Damage

Agentic AI in India: Adoption Is Running Ahead of Control