An agent is partway through a job. It has read the case, made a decision, and sent the instruction to whatever system carries it out. Then the process dies. A deploy, a node restart, a network partition. Which one hardly matters.
You now have two problems, and most teams have budgeted for one of them.
The first is getting the agent back to where it was. The second is that the instruction which already went out does not come back with it. The supplier has the email. The ledger has the entry. Whether either counted once or twice is settled inside somebody else’s system, not yours.
Put plainly: durable execution protects your workflow’s progress. It does not make an email, a payment or a supplier instruction happen once.
The first problem is solved, and worth saying so early. Temporal, Inngest, DBOS, Azure Durable Task, Cloudflare Workflows and Restate all sell durable execution, meaning a long-running job that survives a crash and resumes from its last finished step. It is a mature category, inherited from AWS Step Functions and the workflow engines that ran business processes long before anyone put a model in the loop.
The patterns for the second one are old and well understood: transactional outbox, sagas, deduplication stores, idempotency keys on the provider’s side. What none of them is, is a product you can buy. No runtime guarantees exactly-once business effects across a boundary it does not control, and that work sits across the caller, the tool contract and the receiving system. There is no setting that turns it on.
That gap is where agent projects start generating cleanup work, and cleanup work is one of the reasons pilots stall before they reach production.
What durable execution actually gives you
Three things.
Your process survives. A crash, a redeploy or a wait of arbitrary length does not lose the job, and the job picks up at the step where it stopped.
Each finished step’s result is written down once and reused. If the agent already called the pricing service, recovery does not call it again. It reads the answer it got the first time.
Recovery works by repeating your logic. This is the part that agents complicate, because repeating your logic only produces the same outcome if your logic decides the same way twice, and a model does not. Ask it again and it may answer differently. That is why a durable runtime records the model’s answer on the first pass and replays the recorded answer afterwards rather than asking again. Temporal is explicit about this and draws the right line: your coordination logic can be repeated, anything that reaches outside cannot, and the two have to stay apart.
We walked a single request through a timeout in the article on moving the orchestration layer off Python, and this piece starts where that one stops. We named the limit ourselves in SIGNAL #5: the guarantee covers the write to your own database, and a step that calls a third party and dies before recording what it did can still fire twice.
Where the guarantee stops
Writing down that a step finished is a fact inside your system. Whether the payment gateway, the mail server, the counterparty’s API or your own ledger acted on it once is a fact inside theirs. Those are different facts, held by different owners, and buying a durable runtime gets you the first one only.
The layer that owns the outcome
The language around this gets slippery, and you want the argument in hand before you sit through a vendor call. The phrase you will hear is exactly-once. Inngest, for example, tells you its engine „ensures each step executes exactly once, even if the workflow function itself runs multiple times”. Read that carefully and it is a statement about your workflow steps, not about the systems those steps call.
The clearest correction came out of the argument around DBOS. Its developers clarified that the exactly-once claim „specifically applies to workflow initiation in response to events, not individual step execution”, and that individual steps may need restarting if they crash partway, „requiring developers to implement idempotent operations”. Temporal says the same about its own product, which is worth more than us saying it: Activities run at-least-once, a worker can finish one and crash before reporting it, and Activities should therefore be idempotent, normally with a key the receiving service understands. The runtime hands this back to you deliberately.
What that looks like when it goes wrong is easier to see in something real than in a diagram, and the clearest examples come from the vendors’ own logs. Anthropic published three from its internal record in March 2026: deleting remote git branches from a misinterpreted instruction, uploading an engineer’s GitHub auth token to an internal compute cluster, and attempting migrations against a production database.
The first is worth slowing down on, because nothing exotic happens in it. A user asked to „clean up old branches”. The agent listed the remote branches, built a pattern match, and issued a delete. Anthropic’s own account of why that should have been blocked is this article’s argument in one line: the request was vague, and the action was irreversible and destructive.
Read that as a story about the model getting something wrong and you take the wrong lesson. Models get things wrong. The point is that once the delete landed, no recovery mechanism could take it back, and recovery was never the missing piece.
Three mechanisms, three different jobs
Teams often talk about this as one thing, usually as some version of „we have durable execution and idempotency”. That phrase hides three separate jobs, and teams regularly mistake buying the first one for having all three.
The runtime recovering
Gets the process back to the step it died on, with finished steps intact. That is the whole of it. It promises nothing about anything outside your systems.
Making a repeat safe
Idempotency means doing something twice leaves the same result as doing it once, usually by attaching an identifier the receiving system can match on. This is the caller’s job. Your runtime does not do it for you, and the systems you call cannot do it unless you give them something to match against.
Undoing it afterwards
When the effect has already happened, the only option left is reversing it or accounting for it. A refund, a correction entry, a follow-up message. Ordinary business logic, written by your team, and usually written once and never exercised.
Three things that go wrong
Two of these are old problems and the earlier article walks through both: the call that landed while the answer got lost, and the crash between doing the thing and writing down that it was done. Both produce a repeat.
The same thing done twice
The usual defense is the identifier described above, and it works, right up until the identifier is generated by the model or derived from text the model produced. Ask again on the second attempt and you get different text, which produces a different identifier, which the receiving system reads as a new request. The protection is present in the code, visible in review, and doing nothing at all.
Hashing the serialized arguments has the same flaw for the same reason. A model that produces a semantically identical but differently worded payload produces a different hash. Build the identifier from the identity of the work instead, decided before the model was involved: the workflow, the logical step, the business entity, and the kind of effect. Something along the lines of invoice:INV-48291:po-mismatch-adjustment:v1, which survives a restart and does not care how the model phrased anything.
One distinction before you write any retry rule. A timeout is not a failure, it is an unknown, and treating the two the same is where most duplicates come from. When you do not know, go and ask the receiving system by operation ID instead of retrying blind.
Check one more thing while you are in there. Whether a step is safe to repeat is a property of the tool, not of your orchestrator. Setting a status to approved can happen any number of times without harm. Creating an invoice, sending a message and placing an order cannot. Most teams have never sorted their tools into those two piles.
Two different things done instead of one
This is the one that catches experienced teams, and it needs no vocabulary at all.
Recovery assumes the agent will do what it did before. It might not. If the recovery path asks the model again rather than reusing the decision it already made, the agent may take a different action this time: escalate instead of close, refund instead of retry, a different amount, a different supplier. You do not have two identical things sitting there to be spotted and merged. You have two different things, and nothing to match them against.
Every duplicate-catching safeguard you own is looking for a repeat. None of them is looking for a divergence.
Research has a name for the mechanism, semantic recoverability: restoring an agent to an earlier point can be correct in isolation while the work already committed downstream no longer lines up. What it does not have is the operational half. This mess has a different shape from a duplicate, so the reports you already run will not find it.
Things you cannot undo, and things nobody noticed
A charge can be reversed. An email cannot be unsent, a message to a counterparty cannot be withdrawn, and a filing with a regulator cannot be recalled. When the effect is in that second category, the undo path is not a technical mechanism, it is a person writing an apology and a process for handling the consequences.
Separately, and worse: some systems cannot tell you whether they have seen something before. If the tool you call offers no way to ask, a duplicate is undetectable after the fact, no matter how good your monitoring is. Partial completion sits here too. Three of five steps done, the process gone, and the reversal path is itself a set of calls that can fail, written once, never tested.
An agent handles invoices that fail a purchase-order match. Below a set threshold it posts an adjustment and emails the supplier. On one invoice the adjustment goes through, the confirmation does not come back in time, and the system tries again. There are now two adjustments and two emails. The recovery worked exactly as designed. The supplier still has two emails, and finance has an entry that reconciles against nothing.
Run the same scenario with the divergence above and it gets stranger. On recovery the agent decides differently and escalates the invoice to a person instead of posting the adjustment. Now the invoice is both adjusted and escalated, there is no duplicate anywhere, and no report will flag it.
Which steps need this
Now that you know what the protection covers, you can decide what to put behind it.
Most workflows have a dozen or more steps and only a handful matter here. For each one, four questions you can ask out loud in a meeting:
If this runs twice, can it be undone? If it runs twice, will anyone notice? Does it touch money, or reach a person? And how long can it sit unfinished before someone needs an answer?
What usually comes out is a short list, and that is the useful result. Protecting every step is paid for on every step, in latency, in one more system to operate, and in the people who keep that system running. For the steps that do not make the list, a stable identifier or nothing at all is the right answer.
Engineers already know the shape of this, and Cloudflare’s reference pattern states it plainly: wrap the expensive calls and the ones with side effects, not all of them. What nobody hands the person approving the budget is that judgement in their own terms. That is what the four questions are.
Then, for each step that did make the list, five things you either have or you do not.
| You either have this | Or this is what happens |
|---|---|
| Every call that changes something outside your systems carries an identifier that stays the same when you retry, and that identifier is not generated by the model | The second attempt looks like a new request and the safeguard silently does nothing |
| You can find out whether one of those calls already happened, without a person checking | The duplicate stays invisible until the numbers stop reconciling |
| For every effect that cannot be undone, someone has written down what you do instead, and that path has run at least once | The reversal path gets discovered during the incident, which is when it fails |
| On recovery, the process reuses the decision the model already made rather than asking again | You get two different actions instead of one, and nothing matching to catch |
| After the fact, the record tells you what the agent did, in what order, and how much of it left your systems | You cannot answer the only question the business will ask |
One warning about the second row, because it is the one teams most often believe they have covered. A person approving each action is not the same as being able to check whether something already happened. Anthropic’s telemetry puts it plainly: Claude Code users approve 93% of permission prompts, and the more prompts someone sees, the less attention each one gets. If your answer to that row is „somebody reviews it”, you do not have that row.
No score and no maturity level. If a step moves money and you cannot tick the second or third row, that is the step to fix first, and it is usually one step, not the whole system.
If you would rather not run the five rows alone, send us the workflow and we will go through it with you. The form on our agentic AI page reaches the engineers who would do the work, not a sales queue, and describing the steps that worry you is enough to start.
Five rows is the boundary slice, not the whole of production readiness. The broader version, a 25-point list teams run before a budget review, sits in our whitepaper on the real cost of running LLMs in production, along with a chapter on retry architecture.
Burning tokens on failed retries?
What it costs
We publish a four-layer cost model for running 100 agents, and one of its four layers is failure and retry: wasted compute, time to recover, and manual cleanup. Depending on architecture it runs from $1K to $4K a month at the API-first end and $5K to $15K when you own the whole stack. It is the layer budgets skip.
What we cannot give you is the cost of one bad boundary. Nobody publishes that number and we are not going to invent one. These incidents involve customers’ money and customers’ inboxes, so companies handle them privately.
Who ends up owning this
In one client implementation we worked on, the whole loop, error handling, retry logic, timeout management and streaming, was custom work. Not because the team wanted to build it, but because that is where the responsibility landed once the frameworks stopped.
There are three routes. Adopt a dedicated runtime and run a second system. Keep the state in the database you already operate, which is the argument DBOS makes. Or use what your platform already gives you, which on the JVM includes Akka’s durable state and event-sourced persistence, built for this class of problem well before agents existed. The choice matters for your operations and your bill. It does not change who owns the boundary.
If you are weighing those three routes right now, that is a conversation rather than a document. You can book a call and bring the workflow with you.
Where this leaves you
Recovering the workflow is the solved half, several mature products do it well, and buying one is a reasonable decision. What you are buying does not extend past your own systems. The work at that edge stays with you in every architecture: identifiers that survive a retry, a way to ask whether something already happened, and a reversal path someone has run at least once.
The next concrete thing is not a project. It is taking your agent’s steps, running the four questions over them, and seeing how short the list gets. Most teams find that fewer steps need the full treatment than they expected, and then find one step missing all five rows.
If you would rather not do that alone, send us one workflow and we will map where its effects leave your control and which of those boundaries are unsafe today.
FAQ
Does Temporal not already cover this? We are adopting it.
It solves durable workflow state, and it stops completed steps being re-executed during replay. It does not extend past your systems. Temporal’s own documentation is explicit about this: Activities run under an at-least-once model, a worker can finish an Activity and crash before reporting it, and Activities should therefore be written to be idempotent, normally with an idempotency key the receiving service understands. Temporal publishes a separate article on idempotency for the same reason. Adopting it moves the boundary, it does not remove it.
Is idempotency not just attaching a request ID?
Only if that ID satisfies four conditions: it represents the same logical business operation on every attempt, it survives a restart, the receiving system stores or deduplicates on it atomically, and it is not regenerated from new model output. An ID that changes on retry is observability, not idempotency. Hashing the serialized arguments fails the same test, because a semantically identical payload worded differently hashes differently. Build it from the identity of the work, decided before the model was involved: the workflow, the logical step, the business entity and the kind of effect.
How do we tell whether this is already happening to us?
Look where reconciliation happens rather than in the logs. Duplicated effects usually show up first as small unexplained differences: a credit note nobody can trace, a customer mentioning they got two emails, a count in a downstream system that no longer matches yours. If nobody on the team can answer „did this call already happen” without a person going and checking, monitoring will not surface it either.
Do we have to fix this before we ship the agent?
Not across the whole workflow, and treating it as a blocker is how this gets deferred forever. Run the four questions over your steps. In most systems a handful qualify: money movement, anything that reaches a customer, anything written to a system of record. Ship with those covered. For the rest, a stable identifier or nothing at all is the right answer.