How to evaluate an AI agent: Test sets, metrics, and regression acceptance methods
An agent’s output is inherently uncertain, and a single successful demonstration does not prove that it is ready for production. Effective evaluation should turn “whether the task was actually completed” into a repeatable assessment standard and rerun validation after every change to the configuration, model, or knowledge.
Define “task success” before choosing metrics
Success criteria for the same agent vary across business scenarios. A customer service task may require citing the correct policy and routing the request appropriately; a data task may require read-only queries and correct fields; an automation task must verify the final state rather than merely relying on the model’s claim that the task is “complete.”
| Dimension | Question to answer | Example records |
|---|---|---|
| Task completion | Was the result actually delivered? | Success, failure, human takeover |
| Output quality | Are the facts, completeness, and format acceptable? | Rule-based validation, human scoring |
| Operational efficiency | Is the workflow stable? | End-to-end latency, retries, tool failures |
| Resource cost | Is it affordable at scale? | Model and tool usage per task |
Test sets should cover normal, edge-case, and adversarial inputs
Benchmark test sets should be versioned and include expected results or deterministic evaluation rules. Golden test cases should cover frequent, standard tasks; edge cases should include missing fields, empty results, and API timeouts; adversarial cases should verify how the system handles ambiguous instructions, unauthorized requests, and malicious input. All test data must be de-identified.
Do not leave quality evaluation entirely to another model
Deterministic requirements should be checked with programmatic rules whenever possible, such as a JSON schema, amount calculations, database row counts, and permission states. Model-based scoring can be used for open-ended text, but the scoring criteria should be retained and calibrated through human spot checks. High-risk results must not be approved solely on the basis of model-to-model evaluation.
Integrate evaluation into every change
- Record the versions of the agent, prompts, model, knowledge base, and skills.
- Run the candidate and baseline versions against the same test set.
- Compare failed cases, quality, latency, and resource consumption rather than looking only at the overall score.
- Set business-acceptable thresholds for key metrics.
- Retain reports and failed samples, then rerun the tests after fixes are made.
Real production tasks should also be sampled continuously to identify new inputs not covered by offline testing. SmaugBrain can serve as the platform for running and recording tasks, but the evaluation criteria should still be confirmed by the business owner.
What a one-page acceptance record should include
- The version and reason for the change.
- The test set version and sample scope.
- The number and causes of each type of failure.
- Differences from the current production baseline.
- Human spot-check conclusions and the approver.
- A clear decision on whether to deploy, roll out gradually, or roll back.
Frequently asked questions
How many samples are needed for evaluation?
There is no fixed number that applies to every scenario. Start by covering critical paths and high-risk edge cases, then expand the sample based on the failure rate, business volume, and required confidence level.
Is full testing required for every update?
High-risk changes should undergo full regression testing. For low-risk changes, a critical subset can be run first, but all affected paths must still be covered before deployment.
Can model-based scoring replace human evaluation?
Not entirely. It is suitable for expanding coverage, while humans are responsible for calibrating standards and reviewing edge cases and high-risk results.
What to do next
Start by building a small benchmark set for your most important business workflow, and retain complete version and execution records in SmaugBrain. Once the standards are stable, expand to production sampling and continuous regression testing.