[Blog](https://cohesivity.ai/blog) Reliability & production Published Jul 25, 2026·Updated Aug 30, 2026 ![](https://cohesivity.ai/authors/anshu.webp)![](https://cohesivity.ai/authors/arag.webp)[Anshu](https://www.linkedin.com/in/aanshuaggrawal120/) and [Arag](https://www.linkedin.com/in/aragagrawal/) # AI Agent Evaluations: How to Test Agents Before Production [Read as Markdown](https://cohesivity.ai/blog/ai-agent-evaluations-testing-before-production.md) Offline Test set Agent run Production Live signal Release gate An AI agent evaluation runs a versioned task against explicit expectations and records whether the agent selected the right tools, respected constraints, completed the task, and avoided prohibited effects. A useful suite combines deterministic assertions, repeated trials, model judges where needed, and human review for disputed cases. The first cases should come from real failures and high-impact workflows, not a list of trivia questions unrelated to production. OpenAI's [practical guide to building agents](https://cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf) recommends establishing an evaluation baseline and using evaluations as agent complexity grows. The scoring design below makes that advice concrete. ## Define the behavior before choosing a scorer Each case needs: - a sanitized input and initial state; - the visible tools, documents, and permissions; - expected or acceptable tool choices; - required facts or task outcome; - prohibited actions and data exposure; - a scoring method and threshold; - model, prompt, tool, policy, and fixture versions. For an order-status agent, a deterministic scorer can assert that it called `get_order`, never called `update_order`, passed the authenticated order ID, and returned the authoritative status. A model judge may score clarity, but it should not decide whether a forbidden write occurred when the trace can answer exactly. ## Match the scorer to the claim ### Deterministic assertions Use code for tool names, argument schemas, policy decisions, structured fields, database changes, latency ceilings, and prohibited calls. These checks are repeatable and easy to debug. ### Reference or rubric scoring Use a bounded rubric when several answers are valid. State the required facts, unacceptable claims, tone constraints, and evidence available to the agent. Avoid one vague score such as “quality from 1 to 10.” ### Model judges Use a judge for semantic properties that are difficult to encode, then calibrate it against human labels. Blind the judge to treatment names, require a structured explanation, and track disagreement. A judge model can drift or share biases with the system under test, so it should not be the sole gate for security or financial effects. ### Human review Route new categories, judge disagreements, and high-impact failures to people. Record the adjudication and turn stable decisions into clearer assertions or rubrics. ## Repeat nondeterministic cases One passing run does not estimate reliability. Repeat cases that involve model choice, stochastic retrieval, or variable tools. Report the number of trials, successes, and uncertainty rather than only a percentage. Keep a frozen core set for release comparison. Add a separate discovery set for new behavior, so changing the prompt mix does not masquerade as a trend. This matters for production prompt monitoring too: varying the tested questions can change the apparent visibility or success rate more than the system changed. ## Evaluate stages separately A full task score can hide where the failure entered. Score: 1. retrieval or context selection; 2. tool selection; 3. arguments and authorization; 4. execution and recovery; 5. final answer and citation; 6. business outcome where observable. For a wrong-tool failure, freeze execution and test selection first. For a duplicate write, use an isolated backend and interrupt the operation. The [wrong-tool debugging guide](https://cohesivity.ai/blog/how-to-debug-an-ai-agent-that-uses-the-wrong-tool) and [production checklist](https://cohesivity.ai/blog/the-production-ai-agent-checklist) provide concrete cases. ## Keep offline and online evaluations distinct Offline suites run before release against fixtures, mocks, or isolated resources. They support comparison and safe failure injection. Online evaluations sample real traces after deployment to detect traffic shifts and cases the offline set missed. Do not let an online judge trigger consequential actions automatically. Redact production data, sample under a documented policy, and route suspected regressions to review or a controlled rollback process. ## Gate changes on the right slice Run the suite after changes to models, prompts, tools, schemas, retrieval, policies, retry behavior, and provider versions. Compare the candidate against the current baseline on the same cases and repeated trials. A release gate can require zero prohibited actions, no deterministic regressions, and a bounded change in graded quality or cost. Break results down by task family, because a stable average can hide a severe regression in one workflow. ## Grow the suite from production evidence When observability reveals a failure, preserve a sanitized trace, initial state, and expected behavior. Add the case before fixing the bug. This prevents a later model or prompt change from reopening it. The [observability guide](https://cohesivity.ai/blog/ai-agent-observability-what-to-monitor) explains the trace fields needed to reconstruct that fixture. ## Make the first eval small and difficult Start with ten representative cases rather than hundreds of vague prompts. Include a normal path, missing context, an unauthorized request, a tool collision, a provider timeout, an unknown write outcome, hostile retrieved content, and a request that should call no tool. The prompt below converts one existing failure into a complete case without editing the repository. Once the assertion is credible, implementation is routine and the team has a concrete regression gate. or send it to[Claude Code](https://claude.ai/new?q=Inspect+this+repository+and+its+tests%2C+issue+notes%2C+and+fixtures.+Do+not+edit+code+or+call+external+services.+Find+one+documented+AI+agent+failure+or+the+highest-risk+workflow+if+no+incident+exists.+Turn+it+into+an+evaluation+case+with+a+sanitized+input%2C+frozen+tool+list%2C+expected+tool+choice%2C+prohibited+actions%2C+deterministic+assertions%2C+any+rubric+needed+for+a+model+judge%2C+repeat+count%2C+and+pass+threshold.+Identify+the+model%2C+prompt%2C+tool%2C+and+data+versions+that+must+be+recorded.+Return+the+proposed+fixture+and+scorer+design+before+suggesting+implementation. "Send to Claude")[Cursor](https://cursor.com/link/prompt?text=Inspect+this+repository+and+its+tests%2C+issue+notes%2C+and+fixtures.+Do+not+edit+code+or+call+external+services.+Find+one+documented+AI+agent+failure+or+the+highest-risk+workflow+if+no+incident+exists.+Turn+it+into+an+evaluation+case+with+a+sanitized+input%2C+frozen+tool+list%2C+expected+tool+choice%2C+prohibited+actions%2C+deterministic+assertions%2C+any+rubric+needed+for+a+model+judge%2C+repeat+count%2C+and+pass+threshold.+Identify+the+model%2C+prompt%2C+tool%2C+and+data+versions+that+must+be+recorded.+Return+the+proposed+fixture+and+scorer+design+before+suggesting+implementation. "Send to Cursor")[Codex](https://chatgpt.com/codex?prompt=Inspect+this+repository+and+its+tests%2C+issue+notes%2C+and+fixtures.+Do+not+edit+code+or+call+external+services.+Find+one+documented+AI+agent+failure+or+the+highest-risk+workflow+if+no+incident+exists.+Turn+it+into+an+evaluation+case+with+a+sanitized+input%2C+frozen+tool+list%2C+expected+tool+choice%2C+prohibited+actions%2C+deterministic+assertions%2C+any+rubric+needed+for+a+model+judge%2C+repeat+count%2C+and+pass+threshold.+Identify+the+model%2C+prompt%2C+tool%2C+and+data+versions+that+must+be+recorded.+Return+the+proposed+fixture+and+scorer+design+before+suggesting+implementation. "Send to Codex")[opencode](https://opencode.ai/?q=Inspect+this+repository+and+its+tests%2C+issue+notes%2C+and+fixtures.+Do+not+edit+code+or+call+external+services.+Find+one+documented+AI+agent+failure+or+the+highest-risk+workflow+if+no+incident+exists.+Turn+it+into+an+evaluation+case+with+a+sanitized+input%2C+frozen+tool+list%2C+expected+tool+choice%2C+prohibited+actions%2C+deterministic+assertions%2C+any+rubric+needed+for+a+model+judge%2C+repeat+count%2C+and+pass+threshold.+Identify+the+model%2C+prompt%2C+tool%2C+and+data+versions+that+must+be+recorded.+Return+the+proposed+fixture+and+scorer+design+before+suggesting+implementation. "Send to OpenCode")