Evaluating AI Agents Beyond ‘Looks Good’

An agent can produce a convincing answer and still be wrong in the ways that matter. The difficult part is turning ‘this seems good’ into an evaluation process that a team can run repeatedly and trust.

A good demo is not an evaluation

The first version of an AI feature is often evaluated conversationally. Someone tries a few prompts, reads the responses, and decides whether the behavior feels useful. That is a reasonable way to explore an idea, but it is a weak way to decide whether a system is ready to change.

The same response can look excellent to one reviewer and incomplete to another. A prompt change may improve the examples everyone remembers while breaking less visible cases. Even when the output is fluent, it may ignore a constraint, use the wrong evidence, or reach the right conclusion for the wrong reason.

The problem is not that human judgment has no value. It is that informal judgment is difficult to reproduce. If a team cannot explain what success means and run the same challenge again, it cannot distinguish improvement from a persuasive anecdote.

Start with expected outcomes, not exact sentences

Traditional tests work well when one input should produce one precise output. Agent responses are usually more flexible. Two answers can use different words and still satisfy the same user need, while a textually similar answer can miss a critical requirement.

I find it more useful to define an expected outcome as a small behavioral contract. It describes what the response must accomplish, what it must avoid, and which facts or actions are essential. The contract should be specific enough to grade but broad enough to allow more than one valid expression.

• The essential facts, decisions, or actions the response should contain.
• Constraints it must follow, such as refusing unsupported assumptions.
• Evidence or tool results it should use when they are available.
• Failure behavior when information is missing or a dependency is unavailable.

Use more than one kind of evaluator

No single evaluator is reliable for every property. Deterministic checks are excellent for facts that can be expressed as rules: required tool calls, valid JSON, prohibited content, exact identifiers, latency limits, or whether a workflow reached a known state. They are fast and easy to debug, but they cannot judge every semantic quality.

A model-based judge can compare a response with an expected outcome and assess qualities such as completeness, relevance, or whether the reasoning is supported. That flexibility is useful, but it introduces another probabilistic system. The judge needs a clear rubric, constrained output, and its own calibration against examples reviewed by people.

Human review remains valuable for ambiguous cases and for checking whether the evaluation itself reflects the product. The goal is not to remove people. It is to use their attention where judgment is genuinely needed instead of asking them to reread every routine result.

A score should explain what failed

A single aggregate score is attractive because it makes comparison easy. It is also dangerous because two systems can receive the same score for completely different reasons. One may be accurate but verbose; another may be concise while omitting essential actions.

Useful evaluation output preserves dimensions. Correctness, completeness, groundedness, instruction following, tool use, and safety may not all apply to every agent, but the dimensions that do apply should remain visible. A release decision becomes much clearer when the team can see which behavior moved and which scenarios regressed.

The result should also retain the evaluator's reason and the specific expected outcome that was missed. Evaluation is most valuable when it shortens investigation, not when it merely produces a dashboard number.

Evaluate the evaluator

An LLM judge can prefer polished language over correct behavior, reward answers that resemble the reference too closely, or become inconsistent when the rubric is vague. It may also share blind spots with the system it is judging.

Before trusting a judge, I would compare it against a deliberately varied set of human-reviewed examples: clearly good responses, clearly bad ones, plausible near misses, and cases where multiple answers are acceptable. Disagreements are not just noise. They reveal where the rubric is underspecified or where a deterministic check would be safer.

• Keep judge inputs focused and remove irrelevant context.
• Require structured decisions rather than free-form impressions.
• Track agreement with reviewed examples over time.
• Version prompts, rubrics, test cases, and model configuration together.

Make evaluation part of the development loop

Evaluation becomes much more useful when it runs before a change reaches users. A developer should be able to compare a candidate prompt, model, tool definition, or workflow against a stable suite and inspect the regressions. The suite then becomes a form of executable product knowledge.

In previous work, I created a reusable evaluation repository using an LLM-as-a-judge approach and defined expected outcomes. It eventually supported agent testing across more than 14 teams. The reusable part mattered: teams could share the mechanics while retaining scenarios and rubrics that matched their own products.

A practical suite should remain small enough to run frequently and diverse enough to represent important behavior. Production failures, support cases, and newly discovered edge cases can become regression scenarios. Over time, the suite should grow because the team learned something—not because a large test count looks impressive.

What evaluation cannot prove

A passing suite does not prove that an agent is universally reliable. It shows how the system behaved on a defined set of scenarios under a defined evaluator. Real inputs will be messier, dependencies will fail, and user intent will exceed the boundaries of the test set.

That is not a reason to avoid evaluation. It is a reason to state what the evidence covers. Offline evaluation, staged release, production monitoring, user feedback, and incident review answer different questions. Reliable AI systems need those layers to work together.

What I took away

• Define success as expected behavior, not an exact reference sentence.
• Combine deterministic checks, model judges, and focused human review.
• Preserve evaluation dimensions and reasons instead of hiding them behind one score.
• Calibrate and version the evaluator as carefully as the system under test.
• Treat evaluation as recurring engineering infrastructure, not a one-time launch gate.