[Blog](https://cohesivity.ai/blog) Reliability & production Published Aug 1, 2026·Updated Aug 30, 2026 ![](https://cohesivity.ai/authors/anshu.webp)![](https://cohesivity.ai/authors/arag.webp)[Anshu](https://www.linkedin.com/in/aanshuaggrawal120/) and [Arag](https://www.linkedin.com/in/aragagrawal/) # The Production AI Agent Checklist: Infrastructure, Security, and Reliability [Read as Markdown](https://cohesivity.ai/blog/the-production-ai-agent-checklist.md) Access Identity Permission Runtime Tool path State Before an AI agent touches production data, give it a scoped identity, make every retryable write idempotent, and stop destructive or financial actions before execution. Before it runs unattended, add bounded execution, end-to-end traces, and a tested recovery path. This checklist is ordered by dependency. Later controls rely on the earlier ones. It is not a probability ranking, because no public dataset establishes how often each failure occurs across production agents. ## Before the agent touches production data ### Give each agent a scoped identity - Issue a separate credential for each agent, deployment, or session that needs attribution. - Limit it to the tenant, resources, and actions required for the task. - Enforce authorization in the downstream service, even when the tool description says the action is read-only. - Make credential revocation independent from deleting the human user's account. A shared administrator key makes a clean audit trail impossible and turns one leaked secret into broad access. OWASP's [Excessive Agency guidance](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/) recommends minimizing tool functionality, permissions, and autonomy, then enforcing authorization in downstream systems. ### Stop high-impact actions before execution - Require human or policy approval for destructive, financial, and unusually broad operations. - Show the exact resource, scope, and consequence at the approval step. - Prefer reversible state changes, such as soft deletion, when the underlying system supports them. - Keep audit logs, but do not treat a record written after deletion as a preventive control. Railway documented an April 2026 incident in which an agent found an account-scoped token on disk and called a legacy volume deletion API. The platform had four authorization layers, but the token's scope and the immediate-delete behavior still permitted the action. Railway recovered the volume and changed API deletion to a **48-hour soft-delete window**, matching its dashboard. The [incident report](https://blog.railway.com/p/your-ai-wants-to-nuke-your-database) is a concrete reason to pair scoped identity with a reversible or approved destructive path. ### Make writes idempotent - Create one idempotency key for each logical operation and reuse it for every retry. - Store the operation result on the server and return it when the same key appears again. - Reject a reused key when its parameters differ from the original request. - Define how long keys remain valid and what happens after they expire. Stripe's API stores the first result for an idempotency key and permits keys up to 255 characters. Its current documentation says keys can be removed after they are at least 24 hours old. Those numbers are Stripe's contract, not a universal default; your retention window should cover the longest realistic retry or resume period for your agent. See the detailed [agent retry and recovery guide](https://cohesivity.ai/blog/ai-agent-failure-recovery-retries-checkpoints-human-approval) for the surrounding flow. ## Before the agent runs unattended ### Bound each run - Cap tool calls, wall-clock time, model spend, and repeated attempts at the same action. - Stop when the agent reaches a policy limit rather than asking the model to police itself. - Put narrower limits on write tools than on read tools. - Escalate repeated authorization and validation failures instead of retrying them. Rate limits reduce the damage an agent can cause, but they do not correct an unsafe permission. Use them as a containment layer after identity and authorization are in place. ### Trace the model, agent, and tools together Record one trace across the user request, model calls, tool selection, tool execution, and downstream API work. Include the tenant, agent identity, tool name, tool-call ID, outcome, duration, and policy decision. Exclude secrets and sensitive tool results. [OpenTelemetry now defines GenAI agent and tool spans](https://opentelemetry.io/blog/2026/genai-observability/), including tool-call identifiers and duration attributes. Using those shared fields makes it easier to follow one action across model and application telemetry. The [AI agent observability guide](https://cohesivity.ai/blog/ai-agent-observability-what-to-monitor) covers a practical trace shape. ### Test interruption and recovery - Persist a checkpoint after a completed external side effect, not only after a model response. - Resume from the last verified checkpoint rather than replaying the entire run. - Reconcile unknown outcomes before retrying a write. - Practice revoking the agent credential and stopping an active run. An unknown outcome occurs when the server may have completed a request but the client did not receive the response. Recovery must check the operation record or idempotency key before deciding to repeat it. ## Before usage scales ### Keep a repeatable evaluation set Freeze representative tasks, expected tool choices, prohibited actions, and acceptance criteria. Run the set after changing models, prompts, tool descriptions, permissions, or retry behavior. Include failure cases, because a successful demo does not exercise the controls above. ### Write the incident path down The on-call path should answer four questions without relying on the agent that failed: 1. How do we stop new runs? 2. How do we revoke the affected credential? 3. Which resources and operations did the agent touch? 4. How do we recover or compensate for completed actions? If the team cannot answer those questions from telemetry and backend state, the system is not ready for unattended production work. ## Audit one workflow before production Choose the workflow with the broadest permissions. For every write, record its identity, authorization rule, idempotency mechanism, approval requirement, trace field, and recovery action. Any blank cell becomes a concrete task; you do not need another general checklist. | Write action | Identity and scope | Idempotency | Approval | Trace | Recovery | | --------------------------------- | ----------------------- | ------------- | ---------------------------- | ------------------------- | --------------- | | Example: delete a test deployment | deployment-scoped agent | operation key | required outside test tenant | tool call and resource ID | 48-hour restore | Pick the row with the most blanks. That is the workflow to test first. The prompt below creates an isolated Cohesivity tenant, tries one clean run and one interrupted run, and reports the controls that fail. Production is not invited. or send it to[Claude Code](https://claude.ai/new?q=Find+the+highest-risk+AI+agent+workflow+in+this+project.+Audit+every+write+for+scoped+identity%2C+downstream+authorization%2C+idempotency%2C+approval%2C+trace+coverage%2C+and+recovery.+Do+not+touch+production+data.+Run+npx+%40cohesivity%2Finit+to+create+an+isolated+Cohesivity+tenant%2C+provision+only+the+backend+resources+needed+for+the+test%2C+exercise+one+success+and+one+interrupted+run%2C+and+report+each+failed+control+before+asking+me+whether+to+claim+the+project. "Send to Claude")[Cursor](https://cursor.com/link/prompt?text=Find+the+highest-risk+AI+agent+workflow+in+this+project.+Audit+every+write+for+scoped+identity%2C+downstream+authorization%2C+idempotency%2C+approval%2C+trace+coverage%2C+and+recovery.+Do+not+touch+production+data.+Run+npx+%40cohesivity%2Finit+to+create+an+isolated+Cohesivity+tenant%2C+provision+only+the+backend+resources+needed+for+the+test%2C+exercise+one+success+and+one+interrupted+run%2C+and+report+each+failed+control+before+asking+me+whether+to+claim+the+project. "Send to Cursor")[Codex](https://chatgpt.com/codex?prompt=Find+the+highest-risk+AI+agent+workflow+in+this+project.+Audit+every+write+for+scoped+identity%2C+downstream+authorization%2C+idempotency%2C+approval%2C+trace+coverage%2C+and+recovery.+Do+not+touch+production+data.+Run+npx+%40cohesivity%2Finit+to+create+an+isolated+Cohesivity+tenant%2C+provision+only+the+backend+resources+needed+for+the+test%2C+exercise+one+success+and+one+interrupted+run%2C+and+report+each+failed+control+before+asking+me+whether+to+claim+the+project. "Send to Codex")[opencode](https://opencode.ai/?q=Find+the+highest-risk+AI+agent+workflow+in+this+project.+Audit+every+write+for+scoped+identity%2C+downstream+authorization%2C+idempotency%2C+approval%2C+trace+coverage%2C+and+recovery.+Do+not+touch+production+data.+Run+npx+%40cohesivity%2Finit+to+create+an+isolated+Cohesivity+tenant%2C+provision+only+the+backend+resources+needed+for+the+test%2C+exercise+one+success+and+one+interrupted+run%2C+and+report+each+failed+control+before+asking+me+whether+to+claim+the+project. "Send to OpenCode")