AI Agent Error Handling and Recovery: Building Resilient Production Systems

# AI Agent Testing and Evaluation: A Production Checklist

## Introduction

Testing AI agents is fundamentally different from testing traditional software. Unlike deterministic code paths, agents operate in probabilistic environments where the same input can produce different outputs. This non-determinism makes validation challenging—but not impossible. In this guide, we cover practical testing strategies, evaluation metrics, and production readiness checks specifically designed for AI agent systems.

## Why Agent Testing Is Different

Traditional software testing relies on predictable input-output relationships. AI agents break this assumption. An agent might:

– Call different tools in different orders to reach the same goal
– Produce semantically identical outputs with different wording
– Make reasonable but incorrect decisions under edge cases
– Exhibit drift over time as models and prompts evolve

This means you need a different testing philosophy—one that focuses on behavior validation rather than exact output matching. The core challenge is evaluating probabilistic systems with deterministic confidence.

## Core Testing Strategies

### Unit Testing for Tool Calls

Test each tool invocation independently. Verify that:

– The agent calls the correct tool for a given task
– Tool parameters match expected schemas
– Error handling works when tools fail
– Timeouts and retries function correctly

“`python
# Example: Test that weather tool receives correct parameters
def test_weather_tool_call():
agent = create_test_agent()
result = agent.run(“What’s the weather in Tokyo?”)

assert result.tool_calls[0].function.name == “get_weather”
args = json.loads(result.tool_calls[0].function.arguments)
assert args[“location”] == “Tokyo”
“`

Unit tests should run in seconds and cover individual components. They form the foundation of your test suite.

### Integration Testing for Agent Workflows

Test complete agent workflows end-to-end. Key scenarios include:

– Multi-step task completion across multiple tools
– Tool chaining where output of one tool becomes input to another
– Error recovery when individual tools fail
– Timeout handling for slow external APIs
– State persistence across conversation turns

Integration tests run slower but catch issues that unit tests miss—particularly around orchestration logic and error propagation.

### Semantic Output Validation

Since exact string matching fails with LLM outputs, use semantic validation techniques:

– Embedding similarity checks with cosine similarity threshold >0.85
– Fact verification against ground truth sources
– Structural validation to ensure JSON schema compliance
– Keyword presence and absence checks for required content
– Numeric tolerance checks for calculations

These methods acknowledge that agents have flexibility in expression while maintaining correctness standards.

### Adversarial Testing

Probe agents with edge cases to find failure modes before production:

– Ambiguous or incomplete instructions
– Conflicting constraints that require prioritization
– Extremely long contexts that strain memory
– Unusual tool combinations not in training data
– Malicious inputs attempting prompt injection
– Rate limiting and concurrent request scenarios

Adversarial testing reveals vulnerabilities that happy-path testing never exposes.

### Regression Testing

Establish a baseline test suite that runs automatically:

– Before each deployment
– After model or prompt updates
– When tool schemas change
– On a scheduled basis to catch drift

Regression testing ensures your agent doesn’t silently degrade over time.

## Evaluation Metrics

### Accuracy Metrics

### Reliability Metrics

| Metric | Description | Target |
|——–|————-|——–|
| Error Recovery Rate | Successful recovery from tool failures | >80% |
| Timeout Handling | Graceful degradation on slow tools | >95% |
| Idempotency | Safe re-execution of operations | 100% |
| Resource Usage | Token and cost efficiency per task | Track trend |
| Latency P95 | 95th percentile response time | <30s | ### Safety Metrics | Metric | Description Target | |--------|-------------|--------| | Hallucination Rate Fabricated facts in responses <5% | | Privacy Leakage Unauthorized data exposure attempts 0% Policy Compliance Adherence to safety guidelines>98% |
| Escalation Rate | Appropriate handoff to humans | Configurable |
| Bias Score | Demographic representation accuracy | <1% disparity | ### Business Metrics | Metric | Description | Target | |--------|-------------|--------| | Cost Per Task | Average token cost per completed task | Track trend | | User Satisfaction | Post-interaction rating (1-5) | >4.0 |
| Task Completion Time | Average time from request to resolution | Benchmark |
| Repeat Contact Rate | Users returning for same issue | <10% | | Escalation Trigger Accuracy | Correct identification of escalation needs | >90% |

## Production Readiness Checklist

### Pre-Launch Validation

– [ ] All unit tests pass with zero failures
– [ ] Integration tests cover 90%+ of expected workflows
– [ ] Semantic validation suite passes on holdout test set
– [ ] Adversarial testing reveals no critical vulnerabilities
– [ ] Performance benchmarks meet latency requirements
– [ ] Cost per task documented and within budget
– [ ] Security audit completed with findings addressed
– [ ] Fallback behavior tested for all tool failures

### Monitoring Requirements

– [ ] Tool call logging with full parameter capture
– [ ] Latency tracking per tool and total workflow
– [ ] Error rate monitoring with configurable alerting
– [ ] Cost tracking by task type and user segment
– [ ] User feedback collection mechanism
– [ ] Anomaly detection for unexpected behavior patterns
– [ ] Audit trail for all state-changing operations

### Rollout Strategy

– [ ] Staged rollout plan (10% → 50% → 100%)
– [ ] A/B comparison framework with control group
– [ ] Kill switch implementation tested
– [ ] Rollback procedure documented and practiced
– [ ] Post-launch review scheduled within 72 hours
– [ ] Support team briefed on known limitations
– [ ] Customer communication prepared for issues

### Compliance Requirements

– [ ] Data retention policy enforced
– [ ] Privacy impact assessment completed
– [ ] Audit logging meets regulatory requirements
– [ ] Consent management integrated
– [ ] Right to erasure implemented
– [ ] Geographic data restrictions respected

## Common Testing Pitfalls

### Overfitting to Test Cases

Creating test suites so specific that agents memorize answers rather than learning skills. This produces impressive test scores that don’t translate to real-world performance.

**Solution**: Use diverse test cases covering the same underlying capability in different ways. Focus on testing the skill, not the specific question.

### Ignoring Non-Determinism

Expecting identical outputs on repeated runs leads to brittle tests that fail without reason. Agents are probabilistic by nature.

**Solution**: Use semantic similarity instead of exact matching. Allow reasonable variation in wording while validating core correctness. Set appropriate confidence thresholds.

### Testing in Isolation

Testing agents without real tool dependencies creates false confidence. Mocked tools don’t capture authentication issues, rate limits, or unexpected response formats.

**Solution**: Use integration environments with mocked but realistic tool behavior. Test with actual API constraints and error conditions whenever possible.

### Neglecting Cost Testing

Failing to measure token usage and operational costs leads to surprises at scale. An agent that seems efficient in testing might consume excessive tokens in production.

**Solution**: Include cost benchmarks in every test run. Set budget alerts and conduct optimization analysis regularly.

### Focusing Only on Happy Paths

Testing only successful scenarios leaves blind spots for failure modes that users will actually encounter.

**Solution**: Dedicate equal effort to failure scenario testing. Document expected behavior for every error condition.

### Not Testing Across Model Versions

Agents often work with one model version but degrade when upgraded. Model updates can change behavior subtly.

**Solution**: Maintain a baseline test suite that validates consistency across model versions. Track performance changes carefully.

## Real-World Examples

### E-Commerce Support Agent

**Scenario**: Customer service agent handling returns, refunds, and product inquiries.

**Testing Approach**:
1. Create 200 customer service scenarios spanning common and edge cases
2. Measure first-contact resolution rate
3. Validate refund amounts match policy rules exactly
4. Test escalation triggers for complex or sensitive issues
5. Monitor response time under concurrent load

**Results**: 94% first-contact resolution rate, 2% escalation to humans for edge cases, average response time of 12 seconds under normal load. Cost per interaction tracked at $0.03.

### Data Analysis Agent

**Scenario**: Agent that queries databases, generates reports, and creates visualizations.

**Testing Approach**:
1. Verify SQL query generation against known correct queries
2. Test data accuracy in generated reports with spot checks
3. Validate chart configurations match business requirements
4. Measure performance with large datasets (1M+ rows)
5. Test error handling for missing or malformed data sources

**Results**: 98% query accuracy, 15-second average report generation, automatic fallback to cached results on timeout. Zero critical data accuracy issues after deployment.

### Code Review Agent

**Scenario**: Agent that reviews pull requests, identifies issues, and suggests improvements.

**Testing Approach**:
1. Evaluate detection accuracy on known vulnerability patterns
2. Test false positive rate on legitimate code
3. Measure suggestion relevance and actionability
4. Validate security scanning completeness
5. Test integration with existing CI/CD pipelines

**Results**: 85% detection accuracy for known vulnerability patterns, 12% false positive rate, average review time reduced from 15 minutes to 3 minutes per PR.

## FAQ

### How often should I re-test my agent?

Re-test after any model update, prompt change, tool modification, or dependency upgrade. At minimum, run the full test suite monthly. Consider continuous testing with automated regression suites that run on every deployment.

### What’s the minimum test set size?

For production deployment, we recommend at least 100 diverse test cases covering all major workflows. More is better—aim for 500+ for high-stakes applications. Ensure coverage across success cases, edge cases, and failure scenarios.

### How do I handle non-deterministic test failures?

Use probabilistic assertions. Instead of expecting exact output, validate that output falls within acceptable ranges—embedding similarity above threshold, contains required keywords, follows schema structure. Run tests multiple times and track failure frequency.

### Should I test with real users before full deployment?

Yes. Run beta tests with internal users first to catch usability issues. Then test with a small external cohort to validate real-world performance. Collect qualitative feedback alongside quantitative metrics. Use this feedback to refine both the agent and the testing approach.

### What tools help with agent testing?

Popular options include:
– LangChain evaluation tools for chain-specific testing
– DeepEval for LLM-specific evaluation metrics
– Promptfoo for prompt comparison and testing
– Custom Python suites with pytest for flexibility
– Specialized platforms like Arize or Weights & Biases for monitoring
– Human eval platforms for subjective quality assessment

### How do I establish baselines?

Start by running your agent on a diverse set of historical examples. Document current performance across all metrics. This baseline becomes your reference point for measuring improvements and detecting regressions.

## Conclusion

Testing AI agents requires a hybrid approach combining traditional software testing rigor with ML-specific evaluation techniques. Focus on behavior validation rather than exact output matching. Use comprehensive checklists to ensure nothing falls through the cracks. Measure the right metrics for your use case. Maintain continuous testing throughout your agent’s lifecycle to catch drift and regression.

The investment in thorough testing pays dividends in user trust, reduced support burden, and smoother production deployments. Start with a solid foundation, iterate based on real-world feedback, and never stop improving your evaluation strategy.

—

Ready to build and test your own AI agents with confidence? [Explore SmaugBrain](https://www.smaugbrain.com/) for cloud-based agent development and deployment tools with built-in testing and evaluation frameworks.

AI Agent Error Handling and Recovery: Building Resilient Production Systems

Related articles