[Blog](https://cohesivity.ai/blog)

Architecture

Published Jul 17, 2026·Updated Aug 30, 2026

![](https://cohesivity.ai/authors/arag.webp)![](https://cohesivity.ai/authors/shouryamaan.webp)[Arag](https://www.linkedin.com/in/aragagrawal/) and [Shouryamaan](https://www.linkedin.com/in/shouryamaanjain/)

# Durable Execution for AI Agents: Checkpoints, Retries, and Recovery

[Read as Markdown](https://cohesivity.ai/blog/durable-execution-for-ai-agents.md)

Agent run

Checkpoint

Resume

Durable execution lets an AI agent resume a multi-step task after a process crash, timeout, deployment, or long wait without repeating side effects that already completed. It requires persisted progress, repeat-safe writes, and a recovery rule for calls whose outcome is unknown.

A retry loop alone is not durable execution. Retrying an email send or payment after a lost response can create the exact failure the system was trying to recover from.

## Model the workflow around external effects

Consider a workflow that creates an invoice, sends it, and marks the task complete. If the send succeeds but the worker crashes before recording success, the next run cannot safely assume either success or failure.

```mermaid
sequenceDiagram
    participant W as Workflow
    participant L as Operation ledger
    participant E as Email provider
    W->>L: Reserve send_invoice:task_442
    W->>E: Send with operation key
    E-->>W: Response lost
    Note over W: Worker restarts
    W->>L: Read operation status
    W->>E: Reconcile by provider ID or key
    E-->>W: Already sent
    W->>L: Mark completed
    W->>W: Continue from checkpoint

```

The workflow record says what the system intended. The provider or authoritative backend says what happened. Recovery reconciles the two before choosing whether to repeat the call.

## Separate workflows from activities

A workflow coordinates steps, waits, branches, and retries. An activity performs an external effect such as calling an API or writing a file. Keep workflow decisions deterministic when using a replay-based engine, and isolate network calls, clocks, random values, and provider SDKs inside activities.

[Temporal’s durable execution model](https://docs.temporal.io/encyclopedia/durable-execution) records event history and uses deterministic replay to restore workflow state after failure. You can adopt that engine or implement a smaller state machine with an operations table and queue. The requirement is the same: persisted history must drive the resume point.

## Make every retryable write idempotent

Assign one key to one logical effect, then reuse that key across client, worker, and provider retries. Store the first terminal result and reject reuse with different parameters.

Stripe’s [idempotent request contract](https://docs.stripe.com/api/idempotent%5Frequests) stores the status code and body of the first request for a key, including failures. Its retention behavior is Stripe-specific, but the design principle transfers: the server, not the model, decides whether a repeated request is the same operation.

If a provider lacks idempotency support, place the operation ledger around the adapter. This reduces duplicates from your own retries, but it cannot prove a timed-out provider call failed. Add a reconciliation lookup or require human review for an unresolved, high-impact outcome.

## Checkpoint verified facts

A useful checkpoint records completed operation IDs and the next legal transition. Write it after verifying an effect, and make the state transition atomic with your own database write where possible.

Do not checkpoint only model messages. “The email was sent” in conversation history may be a model claim rather than a provider result. Store the provider message ID or a reference to the operation record.

The [persistent state guide](https://cohesivity.ai/blog/persistent-state-for-long-running-ai-agents) separates task progress from memory and business data.

## Decide which layer owns each retry

Model SDKs, HTTP clients, queues, workflow engines, tool servers, and providers may all retry. Their multiplication can turn three configured attempts into dozens of calls.

For each operation, document:

- which failures are retryable;
- which layer performs the retry;
- the maximum attempts and elapsed time;
- the backoff and provider rate-limit signal;
- the idempotency key reused across attempts;
- the terminal state after exhaustion.

Validation and authorization failures should stop. Rate limits may wait. Network timeouts on writes should reconcile before retrying. A model should not improvise those rules from an error string.

## Treat waits as persisted states

Human approval, a scheduled time, a webhook, and a long-running deployment are workflow states rather than sleeping processes. Persist the wait condition and wake-up signal. Make duplicate signals safe and verify that the represented user still has authority when a delayed approval resumes.

Expose cancellation as a state transition. Cancellation should stop future steps, but it cannot erase a side effect that already completed. Define compensation, such as voiding an invoice, separately from cancellation.

## Test the ambiguous window

Success-path tests miss the hardest state: the server may have completed a request, but the client did not receive a response. Inject failure immediately after the external call and before the completion record. Restart on a different worker and verify the outcome is reconciled once.

Also interrupt during approval, during backoff, and after cancellation. Record one trace across the original attempt and resumed run using the same workflow and operation IDs. The [failure recovery guide](https://cohesivity.ai/blog/ai-agent-failure-recovery-retries-checkpoints-human-approval) turns those cases into a full runbook.

The prompt below designs the first interruption test without touching production. If the expected outcome is “the agent will figure it out,” the workflow does not have a recovery contract yet.

or send it to[Claude Code](https://claude.ai/new?q=Inspect+this+repository+and+choose+one+AI+agent+workflow+with+at+least+two+external+side+effects.+Do+not+use+production+data+or+invoke+external+services.+Map+its+checkpoints%2C+idempotency+keys%2C+retry+owners%2C+timeout+behavior%2C+and+operation+records.+Design+a+local+or+isolated+test+that+interrupts+the+process+after+the+first+side+effect+may+have+completed+but+before+the+client+records+success.+State+how+the+resumed+run+reconciles+the+unknown+outcome+and+prove+that+it+cannot+duplicate+the+effect.+Report+missing+controls+before+proposing+code+changes. "Send to Claude")[Cursor](https://cursor.com/link/prompt?text=Inspect+this+repository+and+choose+one+AI+agent+workflow+with+at+least+two+external+side+effects.+Do+not+use+production+data+or+invoke+external+services.+Map+its+checkpoints%2C+idempotency+keys%2C+retry+owners%2C+timeout+behavior%2C+and+operation+records.+Design+a+local+or+isolated+test+that+interrupts+the+process+after+the+first+side+effect+may+have+completed+but+before+the+client+records+success.+State+how+the+resumed+run+reconciles+the+unknown+outcome+and+prove+that+it+cannot+duplicate+the+effect.+Report+missing+controls+before+proposing+code+changes. "Send to Cursor")[Codex](https://chatgpt.com/codex?prompt=Inspect+this+repository+and+choose+one+AI+agent+workflow+with+at+least+two+external+side+effects.+Do+not+use+production+data+or+invoke+external+services.+Map+its+checkpoints%2C+idempotency+keys%2C+retry+owners%2C+timeout+behavior%2C+and+operation+records.+Design+a+local+or+isolated+test+that+interrupts+the+process+after+the+first+side+effect+may+have+completed+but+before+the+client+records+success.+State+how+the+resumed+run+reconciles+the+unknown+outcome+and+prove+that+it+cannot+duplicate+the+effect.+Report+missing+controls+before+proposing+code+changes. "Send to Codex")[opencode](https://opencode.ai/?q=Inspect+this+repository+and+choose+one+AI+agent+workflow+with+at+least+two+external+side+effects.+Do+not+use+production+data+or+invoke+external+services.+Map+its+checkpoints%2C+idempotency+keys%2C+retry+owners%2C+timeout+behavior%2C+and+operation+records.+Design+a+local+or+isolated+test+that+interrupts+the+process+after+the+first+side+effect+may+have+completed+but+before+the+client+records+success.+State+how+the+resumed+run+reconciles+the+unknown+outcome+and+prove+that+it+cannot+duplicate+the+effect.+Report+missing+controls+before+proposing+code+changes. "Send to OpenCode")