An AI agent can be online and still fail its users. It may return a fluent but incorrect answer, call the wrong tool, exceed its cost limit, or require a human to rescue a workflow that appeared automated. That is why production teams need AI agent service level objectives (SLOs) that measure successful outcomes, not just infrastructure uptime.
This guide explains which AI agent metrics matter, how to turn them into service level indicators (SLIs), how to set practical targets, and how to connect observability to release decisions. The goal is a compact operating model that product, engineering, security, and business teams can use together.
Why uptime is not enough for an AI agent
Traditional service monitoring asks whether a request returned, how long it took, and whether the server produced an error. Those questions remain important, but an agent adds a semantic execution layer. A technically successful HTTP response can contain an unsupported claim. A tool call can return 200 while changing the wrong record. A workflow can complete after so many retries that its cost and latency no longer make business sense.
OpenTelemetry defines reliability in terms of whether a service does what users expect, not whether it is merely available. It also distinguishes telemetry signals such as traces, metrics, and logs. For agents, that principle means each run needs both operational evidence and outcome evidence. A useful trace should connect the user request, model decisions, retrieval steps, tool calls, policy checks, retries, final output, cost, and evaluation result under one correlation ID.
The practical question is therefore not “Was the agent up?” but “Did an eligible task finish correctly, safely, quickly, and within budget?” That sentence is the foundation for an agent SLO.
The five SLI families that matter in production
A small set of indicators is more useful than a dashboard containing every available metric. Google’s Site Reliability Engineering guidance recommends choosing a handful of representative indicators based on what users care about and warns that averages can hide tail behavior. For a production AI agent, start with five families.
1. Correct completion rate
Correct completion rate is the percentage of eligible tasks that satisfy an explicit acceptance rule. The denominator must exclude cancelled requests, test traffic, and tasks blocked by policy before execution. The numerator should require the intended business outcome, not simply a generated answer.
Acceptance can be deterministic for structured work: a file exists, a record has the expected value, a schema validates, or a public page returns the required status. For judgment-heavy work, combine rule checks with a reviewed evaluation set and calibrated human sampling. Track the reason for failure so “incorrect,” “incomplete,” “unsafe,” and “unverifiable” do not collapse into one opaque number.
2. End-to-end latency
Measure from accepted request to verified outcome. Include queue time, model inference, retrieval, tool execution, retries, approval waits, and post-action verification. Report percentiles such as p50, p95, and p99 instead of only an average. A reasonable target may differ by task class: an interactive support response and an overnight research workflow should not share one latency objective.
3. Tool execution reliability
Tool success should mean more than a successful transport response. Record invocation success, semantic success, retry count, timeout rate, idempotency protection, and verification success. For a write action, distinguish “API accepted the request” from “the intended state is visible after a read-back.” This distinction prevents false success in scheduled and multi-step workflows.
4. Human intervention rate
Human review is not automatically a failure. It may be a deliberate control for high-risk work. The useful metric is unplanned intervention: the percentage of runs that require rescue, correction, or manual replay outside the designed approval path. Segment it by cause. A rising rate can reveal weak instructions, brittle integrations, permission problems, or a task that should not yet be fully autonomous.
5. Cost per successful outcome
Tokens per request are not enough because failed runs and retries still consume money. Calculate total model, retrieval, tool, and infrastructure cost divided by verified successful outcomes. Also track the distribution and enforce per-run ceilings. A cheaper model is not economical if lower task success causes more retries, escalations, and rework. For a deeper control framework, see SmaugBrain’s guide to AI agent budgets, token limits, and tool quotas.
| SLI family | Example definition | Decision it supports |
|---|---|---|
| Correct completion | Verified successful eligible runs / eligible runs | Release or rollback |
| End-to-end latency | p95 time from acceptance to verified result | Capacity and workflow design |
| Tool reliability | Verified tool outcomes / attempted tool actions | Integration hardening |
| Human intervention | Unplanned rescues / autonomous runs | Autonomy boundary |
| Successful-outcome cost | Total run cost / verified successes | Model and routing policy |
How to define an AI agent SLO
An SLI is a quantitative measure of service behavior. An SLO attaches a target and a measurement window to that indicator. A complete statement identifies the population, success condition, threshold, window, exclusions, data source, and owner.
For example: “Over a rolling 28-day window, at least 97% of eligible invoice-extraction runs must produce a schema-valid record whose sampled fields match the source document, with p95 end-to-end latency below 90 seconds and no policy bypass.” This is stronger than “the agent should be accurate and fast” because the team can implement, audit, and act on it.
Do not copy that threshold blindly. Establish a baseline on representative traffic, identify the user consequence of failure, and set a target that is valuable but achievable. Separate workload classes when their risk or time expectations differ. Keep definitions stable enough for trend analysis, but version them when acceptance logic changes.
Build an observability path from request to outcome
Agent observability works when every run can be reconstructed without exposing sensitive content. Give the run a unique identifier and propagate it across model calls, retrieval, tools, approval events, and verification. Emit structured events with timestamps, component names, model and prompt version identifiers, tool names, attempt numbers, policy decisions, latency, normalized error classes, token usage, and cost.
Use logs for detailed events, metrics for aggregate health, and distributed traces for the end-to-end path. OpenTelemetry describes a trace as the path a request takes through a distributed system and a span as a unit of work within that path. That maps naturally to agent execution: the run is the trace; planning, retrieval, model calls, tool calls, approvals, and verification are spans.
Protect telemetry as production data. Redact credentials and personal data before emission, apply role-based access, define retention periods, and avoid storing full prompts or tool payloads by default. Store hashes, references, classifications, or sampled redacted content when they provide enough diagnostic value.
Turn SLOs into release and incident decisions
An SLO matters only if it changes action. Define an error budget: the amount of unsuccessful or out-of-target work allowed during the window. A 97% correct-completion objective permits 3% of eligible runs to miss the target. Track consumption rate and use it as a control signal.
- Healthy: the budget is being consumed slowly. Continue normal releases and experiments.
- Watch: burn rate is elevated or a leading indicator is deteriorating. Increase sampling, compare cohorts, and pause risky expansion.
- Freeze: the budget is likely to be exhausted within the window. Stop autonomy increases and nonessential releases.
- Respond: a safety event, destructive tool error, or severe correctness regression triggers immediate containment regardless of the aggregate budget.
- Recover: deploy a validated fix, replay only idempotent or explicitly approved work, and require the SLI to stabilize before restoring traffic.
Use both lagging and leading indicators. Correct completion is often known after evaluation or user feedback. Tool timeouts, retrieval miss rate, retry growth, context-window pressure, policy denials, and unusual cost can warn earlier. Alerts should name an owner and a response playbook; otherwise they become noise.
A practical implementation sequence
- Define the user outcome. Write one sentence describing what a successful run changes or delivers.
- Specify eligibility. State which requests enter the denominator and which valid exclusions apply.
- Create acceptance checks. Prefer deterministic verification; add calibrated evaluation and human sampling where judgment is unavoidable.
- Instrument the run. Correlate model, retrieval, tool, approval, and verification spans with structured metadata.
- Baseline by cohort. Measure task type, model route, tool, tenant, risk tier, and version before choosing targets.
- Set a small SLO set. Start with correct completion, p95 latency, tool verification, unplanned intervention, and cost per success.
- Define error-budget policy. Write the release, escalation, rollback, and recovery actions before a breach occurs.
- Test the monitors. Inject controlled failures and confirm that dashboards, alerts, traces, and owners behave as designed.
- Review after change. Re-evaluate thresholds when the task, acceptance criteria, model, tool permissions, or user population changes.
Pair this operating model with regression evaluation before deployment. SmaugBrain’s AI agent evaluation guide explains how test sets, metrics, and acceptance methods can support that gate.
Common mistakes to avoid
Using only model metrics. Token count, model latency, and answer scores do not show whether the workflow changed the right system safely. Measure the whole outcome.
Reporting averages without distributions. Average latency and cost can look stable while a small but important cohort suffers extreme delays or spend. Use percentiles and cohort views.
Mixing policy blocks with execution failures. A correctly blocked request may demonstrate that a control works. Report it separately from an eligible run that failed after acceptance.
Changing evaluators silently. A new rubric or judge model can move the metric even when production behavior is unchanged. Version evaluation logic and back-test it against a stable reference set.
Collecting sensitive telemetry by default. Observability should reduce operational risk, not create a new data store containing secrets and personal information. Design redaction and access controls before scaling collection.
Frequently asked questions
What is the most important AI agent metric?
For most production workflows, it is the verified correct completion rate for eligible tasks. It directly connects system behavior to the user outcome. Latency, tool reliability, intervention, safety, and cost explain whether that outcome is sustainable.
How is an AI agent SLO different from an SLA?
An SLO is an internal or published reliability target. An SLA is an agreement that includes consequences when commitments are missed. Teams can and should use SLOs even when no contractual SLA exists.
Should hallucination rate be an SLI?
It can be a diagnostic metric, but outcome-based correctness is usually stronger. “Hallucination” can be defined inconsistently, while an acceptance rule can specify whether claims are supported, fields match evidence, or the final action is correct.
How often should SLOs be reviewed?
Review performance continuously and review definitions after material changes to tasks, models, tools, permissions, evaluators, or users. A scheduled quarterly review is a useful minimum for a stable production workflow.
Can one SLO cover every agent workflow?
No. Interactive, batch, read-only, and high-risk write workflows have different expectations. Use shared metric templates, but define targets and response policies by workload and risk class.
What should a small team implement first?
Start with a run ID, structured tool and model events, deterministic outcome verification, correct completion rate, p95 latency, and cost per verified success. Add error-budget alerts and richer cohorts after those basics are trustworthy.
Operate AI agents with measurable confidence
Production AI agents need more than uptime dashboards. A compact SLO system aligns user outcomes, system evidence, risk controls, and economic limits. It gives teams a shared answer to three questions: Is the agent working, is it safe to expand, and what should we fix next?
SmaugBrain helps teams run cloud AI agent workflows with observable execution, controlled tool access, repeatable automation, and verification. Explore SmaugBrain to design an agent workflow whose success can be measured from request to verified outcome.