[Blog](https://cohesivity.ai/blog)

Reliability & production

Published Jul 24, 2026·Updated Aug 30, 2026

![](https://cohesivity.ai/authors/anshu.webp)![](https://cohesivity.ai/authors/shouryamaan.webp)[Anshu](https://www.linkedin.com/in/aanshuaggrawal120/) and [Shouryamaan](https://www.linkedin.com/in/shouryamaanjain/)

# AI Agent Observability: What Should You Monitor?

[Read as Markdown](https://cohesivity.ai/blog/ai-agent-observability-what-to-monitor.md)

Agent

Traces

Run state

Alerts

AI agent observability should reconstruct one run across the user request, model calls, tool choices, policy decisions, tool execution, downstream effects, retries, and final response. Logs, metrics, traces, and evaluations each answer a different question, so collecting model text alone is not enough.

Begin with a failed or expensive workflow and ask whether one identifier can follow it all the way to the authoritative backend result.

## Trace the whole decision and execution path

```mermaid
sequenceDiagram
    participant U as User
    participant A as Agent
    participant M as Model
    participant P as Policy
    participant T as Tool
    participant B as Backend
    U->>A: Request
    A->>M: Model call
    M-->>A: Tool selection
    A->>P: Authorize identity and action
    P-->>A: Decision
    A->>T: Tool call with operation ID
    T->>B: Downstream request
    B-->>T: Authoritative result
    T-->>A: Structured result
    A-->>U: Response

```

Use a trace for the run, spans for model and tool work, and an operation ID for a side effect that may outlive or outlast the trace. Carry the tool-call ID into the server when possible. A database row should link back to the operation and trace without storing the full trace itself.

OpenTelemetry publishes [GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) for model, agent, and tool spans. The conventions are still evolving, so pin the version and keep a small internal schema layer rather than scattering experimental attribute names through the application.

## Capture facts that explain a run

### Model spans

Record provider, model, response identifier, latency, token usage, finish reason, retry count, and prompt-template version. Store full prompts only when policy permits it. A hash, redacted sample, or protected reference may be enough for ordinary traces.

### Tool spans

Record the visible tool name and version, tool-call ID, redacted arguments, selection source, duration, result class, retry count, and downstream operation ID. Separate “model selected tool” from “policy allowed tool” and “backend completed action.”

### Policy and approval spans

Record the represented identity, tenant, resource, action, policy version, decision, reason code, and approval reference. Never record access tokens or raw secrets. A denied request is an expected policy outcome, not necessarily an application error.

### Recovery spans

Link resumed work to the original workflow and operation. Record whether the system retried, reconciled an unknown outcome, compensated a completed action, or stopped for human review. The [failure recovery guide](https://cohesivity.ai/blog/ai-agent-failure-recovery-retries-checkpoints-human-approval) describes those branches.

## Use metrics for patterns, not explanations

Aggregate metrics should include:

- successful, failed, denied, and cancelled runs;
- task completion and human-escalation rates;
- model and tool latency by percentile;
- token and provider cost per task and tenant;
- tool selection, validation, authorization, and backend error rates;
- retries, repeated calls, loop stops, and unknown outcomes;
- queue delay, active runs, and checkpoint age.

Avoid labels with unbounded values such as prompt text, user IDs, tool arguments, or trace IDs. Put those in protected logs or traces and keep metric dimensions bounded.

## Keep logs as discrete, redacted events

Logs are useful for deployment changes, policy decisions, operation transitions, provider errors, and cleanup. Use stable event names and structured fields. Do not log the same complete model payload at every layer.

Apply redaction before data leaves the process. Secrets, authorization headers, personal data, retrieved documents, and tool outputs can appear in prompts and errors. Sampling after export does not undo exposure.

## Measure quality with evaluations

Telemetry tells you what happened. It cannot decide whether a response was helpful or a tool choice was correct without an expected behavior. Join production failures and sampled traces into a versioned evaluation set, then run it before changing models, prompts, or tool contracts.

The [AI agent evaluation guide](https://cohesivity.ai/blog/ai-agent-evaluations-testing-before-production) separates deterministic assertions, model judges, repeated trials, and release gates.

## Build alerts around actions a human can take

Alert on conditions with a response: broad authorization failures after a deploy, a spike in destructive-tool calls, missing trace continuity, repeated unknown outcomes, queue age, spend limits, and runs exceeding call or time budgets.

Avoid paging on every model error. Group by provider, model, tool, release, and result class first. A dashboard can show noisy quality movement; a page should point to a specific containment or rollback action.

## Audit one trace before adding dashboards

Choose a real run and follow it manually. Can you identify the prompt version, tool list, policy decision, downstream request, business result, cost, and any retry? Can you find the affected operation from the trace and the trace from the operation?

The prompt below performs that audit against the repository without transmitting data. The first missing link is a better instrumentation task than another aggregate chart.

or send it to[Claude Code](https://claude.ai/new?q=Inspect+this+repository%27s+AI+agent+instrumentation.+Do+not+edit+code+or+send+telemetry.+Choose+one+workflow+that+calls+a+model+and+at+least+one+external+tool.+Map+the+trace+from+user+request+through+model+calls%2C+tool+selection%2C+policy+checks%2C+tool+execution%2C+downstream+API%2C+retry%2C+and+final+response.+List+the+identifiers+and+redaction+rules+at+each+boundary.+Mark+missing+parent-child+links%2C+unbounded+payload+capture%2C+absent+cost+or+outcome+fields%2C+and+places+where+an+operation+cannot+be+connected+to+a+trace.+Return+the+smallest+instrumentation+changes+first. "Send to Claude")[Cursor](https://cursor.com/link/prompt?text=Inspect+this+repository%27s+AI+agent+instrumentation.+Do+not+edit+code+or+send+telemetry.+Choose+one+workflow+that+calls+a+model+and+at+least+one+external+tool.+Map+the+trace+from+user+request+through+model+calls%2C+tool+selection%2C+policy+checks%2C+tool+execution%2C+downstream+API%2C+retry%2C+and+final+response.+List+the+identifiers+and+redaction+rules+at+each+boundary.+Mark+missing+parent-child+links%2C+unbounded+payload+capture%2C+absent+cost+or+outcome+fields%2C+and+places+where+an+operation+cannot+be+connected+to+a+trace.+Return+the+smallest+instrumentation+changes+first. "Send to Cursor")[Codex](https://chatgpt.com/codex?prompt=Inspect+this+repository%27s+AI+agent+instrumentation.+Do+not+edit+code+or+send+telemetry.+Choose+one+workflow+that+calls+a+model+and+at+least+one+external+tool.+Map+the+trace+from+user+request+through+model+calls%2C+tool+selection%2C+policy+checks%2C+tool+execution%2C+downstream+API%2C+retry%2C+and+final+response.+List+the+identifiers+and+redaction+rules+at+each+boundary.+Mark+missing+parent-child+links%2C+unbounded+payload+capture%2C+absent+cost+or+outcome+fields%2C+and+places+where+an+operation+cannot+be+connected+to+a+trace.+Return+the+smallest+instrumentation+changes+first. "Send to Codex")[opencode](https://opencode.ai/?q=Inspect+this+repository%27s+AI+agent+instrumentation.+Do+not+edit+code+or+send+telemetry.+Choose+one+workflow+that+calls+a+model+and+at+least+one+external+tool.+Map+the+trace+from+user+request+through+model+calls%2C+tool+selection%2C+policy+checks%2C+tool+execution%2C+downstream+API%2C+retry%2C+and+final+response.+List+the+identifiers+and+redaction+rules+at+each+boundary.+Mark+missing+parent-child+links%2C+unbounded+payload+capture%2C+absent+cost+or+outcome+fields%2C+and+places+where+an+operation+cannot+be+connected+to+a+trace.+Return+the+smallest+instrumentation+changes+first. "Send to OpenCode")