AI agent failure recovery decides whether a failed step should stop, retry, reconcile, wait for approval, compensate, or resume from a checkpoint. The decision depends on the error class and whether an external effect may already have completed.
Automatic retries are safe only when the operation contract makes repetition safe. A timeout does not prove failure.
Classify the failure before responding
| Failure | Default response |
|---|---|
| Invalid input or schema | Stop and correct the request |
| Authentication or authorization | Stop and request valid access |
| Policy denial | Stop or request the specified approval |
| Rate limit with retry time | Wait, then retry within the operation budget |
| Transient read failure | Retry with bounded backoff |
| Write timeout with unknown outcome | Reconcile before retrying |
| Permanent provider failure | Fail or use a documented alternate path |
| Worker crash | Resume from persisted state |
| Partial multi-system completion | Continue, compensate, or escalate according to policy |
Expose stable error codes from tools. Do not ask the model to infer retry safety from prose such as “something went wrong.”
Give one logical write one operation ID
Create an operation record before a consequential call. Store the action, tenant, resource, normalized input hash, idempotency key, state, attempts, provider reference, and result.
Reuse the same idempotency key for every retry of that logical effect. Reject reuse with different parameters. Stripe’s idempotent request documentation illustrates a server-side contract that returns the stored result for repeated keys. Its retention rules are provider-specific, so set your window from the longest realistic resume path.
If the provider has no idempotency support, your operation table prevents two local workers from intentionally starting the same action. It cannot settle a response lost after the provider acted. Add a provider lookup, webhook, or human reconciliation step.
Checkpoint verified progress
Checkpoint after the system verifies an external effect, not after the model says it happened. Store provider identifiers or operation references. On resume, load the checkpoint and authoritative backend state before calling the model again.
The durable execution guide explains workflow history, deterministic replay, and persisted waits. Failure recovery adds the operational choice for each state.
Assign one retry owner
HTTP clients, queues, workflow engines, model SDKs, tools, and providers may all retry. Choose one owner for each failure class and count every attempt under the same operation.
Set maximum attempts, elapsed time, backoff, and a terminal result. Stop on validation, authorization, and policy failures. For model calls, decide whether a retry uses the same prompt and tools or creates a new decision. Record that distinction because it changes the evaluation.
Reconcile unknown outcomes
When a write times out:
- mark the operation
unknown, notfailed; - query by idempotency key, provider reference, or expected business state;
- mark it completed if the effect exists;
- retry only if the provider confirms it did not occur;
- request human review when neither result can be established.
Do not hide an unknown state behind a success message. The user may need to avoid repeating the request while reconciliation continues.
Put approval before a high-impact attempt
Approval should bind the action, resource, parameters, actor, and expiry. Recheck authorization when execution resumes. A broad approval such as “continue” should not cover modified payment or deletion parameters.
Railway’s agent deletion incident led it to align API deletion with a 48-hour soft-delete period. Reversible effects reduce recovery pressure, but they still require scoped identity and audit records.
Distinguish cancellation from compensation
Cancellation prevents future work. It cannot undo a completed email, payment, or external publication. Compensation is a separate action such as voiding an invoice or restoring a soft-deleted resource.
Compensations need their own authorization, idempotency, operation record, and failure path. Avoid describing them as automatic rollback when multiple external systems lack a shared transaction.
Write a runbook a human can execute
For each high-impact workflow, document:
- how to stop new runs and active workers;
- which credential to revoke;
- how to find tasks and operations in scope;
- how to classify unknown outcomes;
- which retries are safe;
- which effects can be compensated;
- who approves recovery and how completion is verified.
Link the runbook to trace and operation queries. The observability guide lists the identifiers needed to follow an incident end to end.
Practice failure between the lines
Inject failures before a call, after the provider receives it, after it succeeds but before the response arrives, after the checkpoint, during approval, and during cancellation. Resume on a fresh worker with no local state. Verify the effect count and final business state.
The prompt below designs the most revealing drill without touching production. If the runbook cannot distinguish “failed” from “may have succeeded,” fix that branch before increasing autonomy.