This comes up often in my conversations during architecture workshops.
“How do we let LLM Agents touch production systems? How do we mitigate risks when agents would act on their desicions?”
When you sit down with engineering teams to debug what is genuinely stalling deployment, the issue is rarely that the foundation models are not capable enough.
We are continuing our series on what breaks when AI agents transition from prototypes to production:
Part 1: Communication Failure: Why Agent-to-Agent Communication Fails
Part 2: Contract Failure: Why AI Agent Handoffs Fail
Part 3: Context Failure: Your Agent Isn’t Hallucinating…
Part 4: Evaluation Failure: Evaluating AI Agent Handoffs
Part 5: Agent Sprawl: The Hidden Nightmare of Your Next Agent Call
Part 6: Decision Blindness: How to Fix “Decision Blindness” in AI Agents
Part 7: Authority Failure: Stop Giving Agents Tools. Start Giving Them Boundaries.
Part 7: Complexity Failure: The Hidden Cost of Evaluation Debt in Multi-Agent Systems
Today, we look at the final frontier of operational risk: Commitment Failure.
👉 Request: I'd love to hear from you. Take the short survey at the bottom of this article and tell me what you'd like to learn more about.
How an agent creates a duplicate refund
A customer opens the banking app on a Tuesday evening. Salary got credited one day late, and the bank charged a $35 overdraft fee. The customer chats with the bank’s AI assistant.
“I got charged $35 because my salary came one day late. Can you refund it?”
The agent checks the account history and the fee waiver policy. The request looks valid. So it calls a tool:
issue_fee_credit(account_id="9482", amount=35.00)
Now see what happens/ could happen.
The network breaks at the wrong moment. The core banking system gets the call. It credits $35. But the response times out before it reaches the agent.
The agent tries again. The framework gives the timeout error back to the model. The model thinks the credit did not happen. It plans again.
The same write goes twice. It calls
issue_fee_credit(account_id="9482", amount=35.00)one more time.
One hour later, the customer logs in and sees two credits of $35.
For one customer, this is small. Now think of millions of chats where agents have write tools. The same bug can apply a credit limit change two times. It can cancel a subscription that was already cancelled. It can send the same payment twice.
The engineering team checks the logs. Audit trail is clean. Schema validation passed. Auth tokens are fine. Everything looks correct, and still the money went out twice.
Damn!
Answering is not same as acting
The risk of an AI system goes up sharply as it moves through four stages.
Answer: Gives text, summaries and search results. Failure is hallucination or old data. Small blast radius.
Recommend: Gives options, drafts an email, fills a form for a human to check. Failure is bias, or the human approving without reading. Medium blast radius.
Decide: Checks thresholds and triggers rules on its own. Failure is evaluation debt and errors that spread across agents. High blast radius.
Act: Writes to APIs, ledgers and real systems. Failure is duplicate writes, half-done workflows and side effects you cannot undo. Critical blast radius.
Many teams are doing one mistake here. They build the Act stage with the same thinking they used for the Answer stage. They give the model a tool like
issue_fee_credit()and hope normal exception handling will save them.
Why AI agents break the rules idempotency depends on
Look, duplicate writes is not a new problem, distributed systems solved it long back.
The client sends a unique key with every request. The server remembers which keys it has already processed. If a retry comes with the same key, the server returns the old result. It does not run the action again. Payment networks, card schemes and every serious core banking system work like this.
But this works only on one assumption. The retry must come from the same caller, with the same key.
AI Agents break this in three ways.
The model becomes the caller. When a tool call times out, most agent frameworks give the error back to the model. The model plans again and makes a new tool call. New call ID. No memory of the old key. Sometimes even different parameters. The downstream system sees it as a fresh request.
The tool has no natural key.
issue_fee_credit(account_id, amount)talks about an amount. It does not say which fee is being refunded. So nothing downstream can tell a retry apart from a second genuine request. Most duplicate write risk is coming from the tool design itself. The network failure only exposes it.Workflows mix steps you can undo with steps you can’t. Say an agent updates a CRM, a payment platform and an internal ledger. That is a saga pattern, whether anyone designed it like one or not. Saga design has always separated three kinds of steps. Some you can reverse, like a credit. Some you can safely retry. And some you can never take back, like an email to a regulator. These last ones are called pivot steps. Agents get all three as normal tools. Nothing makes the pivot step run last.
So the old principles are still working fine. The problem is where we drew the boundary. We put a model that is not deterministic inside the transaction.
Moving the boundary outside the agent
The fix is simple to say. The agent decides what should happen. A separate, deterministic layer makes it happen, and only once.
Let me go through each part.
1. Design tools around business events, not amounts
The cheapest fix is in the tool schema. Change issue_fee_credit(account_id, amount) to reverse_fee(fee_transaction_id).
One fee can be reversed only one time. So the core system itself can reject the second reversal. The amount also comes from the ledger now, not from the model. That is one less thing the model can get wrong.
If there is no natural business key, the gateway creates one.
2. The harness owns the key and the retry
The gateway creates the idempotency key when the intent is first saved. The model never creates it. Every attempt to the downstream system carries the same key. Retries happen inside the executor, not inside the model’s planning loop.
The executor may send the request more than once. That is fine. The downstream system honours the key, so the action happens once only. This is how you get effectively-once execution.
What the agent gets back is also important. On a timeout, it should get status: PENDING with an intent ID. It should not get an exception. If the model knows the action is pending, there is nothing to plan again. If you give it a stack trace, it will mostly try again.
This also changes what the agent tells the customer. It says “your refund is being processed”. It does not say “done”.
One more thing. Don’t make the key from a hash of the payload. Two genuine requests can have the same payload, for example two different $35 fees on the same account. A hash key will quietly drop the second one. And if a parameter changes on retry, the hash also changes, so the duplicate goes through anyway. Use payload similarity only as an alert for near duplicates. Never as the key.
3. Track the state no one wants to talk about
Every intent goes through clear states. Each change is saved as an event in an append-only log.
STAGED → VALIDATED → SUBMITTED → SETTLED | FAILED | IN_DOUBT
IN_DOUBT is exactly where our refund story landed. The request went out. No response came back. Nobody knows if the core applied it or not. Most agent designs have no state for this. So the model fills the gap by guessing.
How you resolve IN_DOUBT depends on the downstream system.
It supports idempotency keys. Send again with the same key. Worst case, you get the old result back.
It supports lookup by your reference ID. Keep checking the status till you get a clear answer.
It supports neither. This is very common with old core systems. Hold the intent. Block any new intent on the same account. Then resolve it through reconciliation with the system of record, usually the end-of-day ledger match. It is slow. But it is honest.
For every system your agent can touch, you must know which case applies. This belongs in the design review. You don’t want to find it out in the incident review.
4. Label every step before you put it in order
Before you give an agent a multi-step workflow, label each action. Is it compensatable, retryable or pivot?
Then the executor enforces the order. Compensatable and retryable steps go first. The pivot step is released only after everything before it is settled.
The pivot step is where maker-checker comes in. Banks are using this control since decades. The agent prepares the action. A second party approves it before release. That can be a human, or an independent automated check.
Three safeguards to stop Commitment Failure
1. Don’t provide write access for agents
Agents should never write directly to a system of record. They should only create a typed intent. The gateway checks the schema and the policy. A separate executor, with no model inside it, does the actual write.
2. Generate keys from the harness, never from the model
Every intent gets one key when it is created. The same key goes on every retry. If the model tries the same action again, the gateway sees an open intent for the same business entity. It returns the status. It does not create a new write.
3. Send decision to a human for risky actions
Some actions cannot be undone. Large payments, data deletion, production deployments, messages to regulators. For these, hold the action in a pending state. Release it only after a human or an independent control approves.
For lower risk actions, a short hold window can work, say ten minutes. During that window, the action can still be cancelled.
Think about this
Before you allow your next agent to write to production, ask your senior engineers this:
If our agent sends a duplicate or wrong write during a network failure tomorrow, will we catch it and reconcile it before the customer finds it?
If the answer is “the customer will tell us” (lol), then the system is not ready. Right now you are depending on hope.
You have to govern the commitment.
Share your perspective
Take the Reader Survey to shape upcoming deep dives on production agent architectures.
Subscribe to AgentBuild for engineering blueprints delivered weekly.
Share agentbuild.ai with your technical peers.
P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.
Ask your friends to join.
More valuable content coming your way.
The model deciding to retry is the part nobody plans for.
My own logs had 84 browser timeouts in 14 days. The rule I wrote after: never retry navigate after a timeout, look at the page first.