Idempotency Design in Agent Workflow Steps

Agent workflows fail mid-task, all the time, and when they restart they don't remember what they already did. That gap between "the agent didn't get a result" and "the operation never actually happened" is where duplicate charges, double emails, and phantom orders come from. This piece walks through how to design each step so it survives a retry without repeating itself, and what happens when step-level fixes aren't enough for the whole workflow.

Picture a payment API that returns a 200, then the confirmation write fails. The agent sees no confirmation, assumes failure, retries the whole thing. Now there are two charges against one order, and the agent reasoned correctly both times. It just had no way of knowing the first charge landed. This failure mode is not hypothetical — it plays out in production systems where retry logic has no way to check whether the first attempt actually landed.

Multi-step workflows make this worse, not better. If an orchestrator strings together three tool calls and the second one times out, a naive retry re-runs all three. Tool one already sent the email. Tool two already wrote the database row. Now you're not fixing a failure, you're adding a second failure on top of a success. And during a provider outage, when everyone's retrying at once, you're not just risking duplicates, you're burning inference and API spend on work that already happened. Retry storms during incidents can eat a large chunk of a team's monthly inference budget, because every retry re-executes the entire pipeline from scratch.

The instinct is to fix this in the prompt. Tell the agent "only call the payment tool once." That doesn't work, and it's worth saying plainly: the agent that double-charged the customer was following its instructions correctly. It wasn't reasoning badly. It had no external memory of what it had already done. That's not a prompting problem. That's a state problem, and state problems need state solutions.

What idempotency means at the step level, and how it differs from deduplication

Idempotency means calling an operation five times produces the same result as calling it once. Nothing compounds. The classic web example: PUT /users/{id} sets a user's state to some declared value, so calling it ten times leaves you exactly where one call would. POST /orders isn't like that. Ten calls create ten orders. The line is simple once you see it: "set this value to X" is idempotent, "increment this value by one" is not, because the second one's outcome depends on how many times you ran it.

Deduplication is a different tool for a different problem. It's what you use when the operation itself is NOT naturally idempotent, but you still need it to run exactly once. Deduplication works by giving the operation a stable identity, an idempotency key, and checking that key before letting the operation fire again. Without that key, there's nothing to match against, and deduplication has no teeth.

The two concepts aren't interchangeable. An idempotent operation might not need any deduplication machinery at all, it's just safe by design. A non-idempotent operation made safe through deduplication still needs the underlying call itself to be guarded properly. Every tool call in an agent workflow needs to be sorted into one bucket or the other, because the fix looks different depending on which one you're dealing with. According to the AI Agent Idempotent Operations guide from fast.io, LLM agents retry tool calls 15 to 30% of the time, driven by timeouts, validation errors, and plain model uncertainty. At that frequency, an unguarded non-idempotent step isn't a rare edge case. It's a recurring event in any workflow with more than a couple of steps.

Classifying every step in a workflow by its retry risk profile

Not every step deserves the same amount of engineering attention. Four buckets cover most of what shows up in practice.

Read operations are naturally repeatable. Reading a database row twice gives you the row twice, no harm done (freshness and rate limits are separate concerns, but duplication isn't the risk here). Set operations move something toward a declared state, like flipping a feature flag to off, and repeating them is safe as long as the declared value doesn't change mid-flight. Append and create operations are the opposite: by default, each call produces a new artifact, so two calls mean two records. These need key-based deduplication to behave. Then there are irreversible operations: payments, emails, webhooks, deployments. These need the most careful guarding, because a second execution has consequences that don't undo cheaply.

Git is a good mental model for the safe end of this spectrum. Commit identical content twice and you get the same tree SHA both times. Push a branch that's already pushed with nothing new on it, and it's a no-op. File writes behave the same way. Compare that to posting a comment: post it twice, you get two comments, always. Applying a label that's already applied is wasteful but harmless. Same shape of action, completely different risk.

Design effort should go where the risk actually lives. Steps that create records, move money, send notifications, or touch authoritative system state, that's where a partial failure leaves a visible, sometimes expensive, artifact. Steps that are naturally idempotent and don't produce an external record often don't need explicit compensation logic at all. Classification is what tells you where to spend the engineering hours.

Checkpoints help shrink the blast radius. If a workflow captures state before each step, a retry only has to worry about the segment since the last checkpoint, not the whole run. Worth noting: checkpoints typically cover file edits, not shell commands, not branch creation, not external API calls. They narrow the window. They don't close it.

Generating idempotency keys that survive restarts and remain stable across retries

The whole system falls apart if the key itself isn't stable. A key has to come from durable, deterministic state, never from a timestamp, never from a UUID generated fresh at call time. If the key changes between attempts, the deduplication store sees a stranger every time and re-executes the operation, which defeats the entire point.

A construction that holds up combines the workflow run identifier, the step identifier, and the action type into a single composite key. Same run, same step, same action, same key, every single time, whether it's attempt one or attempt five. The key gets regenerated from the same inputs on resume. It's never manufactured new.

Stability actually has to start earlier than the tool call itself. Before any external system gets touched, the workflow needs a stable identifier for the business operation it's about to perform: refund this order, by this amount, create this ticket, for this incident. That identifier is what everything downstream keys off of.

Stripe's API is a useful reference point here, not because it's exotic, but because it's a mature, battle-tested version of exactly this pattern. The core behavior is that the first request tied to a given key sets the result, and any later request with that same key gets that cached result back without re-executing the operation. Keys also carry an expiry, which bounds how long the system has to hold onto old results.

A few guardrails worth building in alongside the key itself. Bind the key to the authenticated user or workflow context, so a captured key can't get replayed by a different actor. Validate timestamps too, and reject anything too old, which stops legitimately expired operations from getting replayed later. And get the order of operations right in the implementation: claim the slot in the deduplication store first, then execute the action. Do it the other way around, execute first and log second, and a crash between those two steps leaves the store with no idea the action ever happened. The next retry walks right past it.

Building the deduplication store that backs every key check

Two layers do different jobs here, and conflating them is a common mistake. Gateway-level handling gives you a fast short-circuit on the happy path, it's a performance optimization, nothing more. Application-level handling is where correctness actually lives: it owns the deduplication store, coordinates multi-step workflows, and makes the real call on whether something already happened. In a working production system, both layers exist, but the gateway is never a substitute for the application layer doing its job.

A common setup pairs a fast datastore, Redis being the usual choice, for key lookups, with a separate database table holding the authoritative idempotency record. That split gives you fast reads without piling extra load onto the system of record.

A three-state model works well in practice: every key gets a status of IN_PROGRESS, COMPLETED, or FAILED, with an expires_at column driving a cleanup job. That cleanup job is about storage housekeeping, not correctness, it's tidying up, not deciding anything.

Atomicity is where a lot of these systems quietly break. The business write and the idempotency status change need to happen inside a single database transaction. If the business write succeeds but the status update fails separately, the whole mechanism is compromised, because now the store doesn't reflect reality.

Retention windows need to match the actual lifespan of the business risk, not just a typical request timeout. If the record expires while the workflow can still legitimately retry, duplication becomes possible again, right when you thought you'd closed the door. For Kafka-based systems specifically, the cleanup window should run well past the worst-case replay window, a minimum of seven days is a reasonable floor, to keep keys from expiring mid-replay.

The failure modes that well-designed key systems still produce in production

Even a carefully built key system breaks in specific, recurring ways once it hits real infrastructure.

Picture a Kubernetes rolling deployment that evicts a pod mid-request. The key got written with status "processing" right before the eviction. The replacement pod has zero knowledge of the original request. Every retry after that finds "processing" sitting there, backs off politely, and the payment fails silently, forever, with nobody the wiser. The fix is to treat "processing" as a short-lived lease, not a permanent marker. If the lease has expired and the key is still stuck on "processing," the next retry should be allowed to reclaim it and try again.

TTL expiry causes a related problem: a key can expire while the client is still inside its retry window, meaning the deduplication coverage ends before the actual risk does. Schema changes cause a subtler one. Change the schema, and a retry's fingerprint no longer matches what's on record, so the store treats it as an entirely different operation and lets it re-execute.

None of these failure modes gets solved by one clever fix in isolation. A key written into an eventually-consistent store can expire before replication finishes across data centers. Using an atomic SET NX EX solves the storage race, but you still need two-phase status handling to deal with partial commits. They interact, and treating them as separate bugs to patch one at a time misses how they compound.

There's also a timing gap in check-before-act patterns worth naming directly: two concurrent runs can both check "does this exist," both get "no," and both go create it. Client-side existence checks fall apart the moment more than one actor can touch the same resource. Server-side idempotency keys with atomic claim semantics are the approach that actually closes this gap, not another existence check.

One more trap: silently skipping an action because state already exists can hide real conflicts. If that pre-existing state came from a different actor, or a stale earlier run, silent success buries the problem instead of surfacing it. Failing loudly on unexpected pre-existing state is often the safer default.

Why step-level idempotency is not enough when steps are sequenced in a workflow

Every step in a sequence can be individually flawless and the workflow can still end up broken. Payment succeeds, confirmation write fails, customer's been charged with no confirmation on file. The agent retries the full sequence. Payment gets deduplicated correctly this time around, good. But the inventory reservation from step two might now fail, because stock hit zero during the first run. Idempotency at the step level did its job and the workflow still ended up in a bad state.

Making each step individually safe to retry doesn't make the sequence atomic. An agent updating a database, sending a message, and publishing an event, guarding each of those three individually doesn't guarantee all three land together as one unit.

This gets sharper in multi-agent setups. Say Agent A orchestrates a chain: retrieve, analyze, write summary, send notification, with a different specialist sub-agent handling each piece. If Agent B's completion signal shows up late, Agent A might retry the entire workflow. Now the notification is in flight a second time, the summary's been written twice, and if that write wasn't guarded properly, there are two records sitting in the database where there should be one.

The fix has to happen at the level of the individual artifact, not the workflow as a whole, because the first run may well have completed several steps successfully before it fell over. Per-step keys are a starting point, not a finish line. What the sequence actually needs is workflow-level state, recovery logic, and in a lot of cases, a way to walk back the steps that already happened. That's a coordination problem, and it needs a coordinator that actually knows what ran and what didn't.

The saga pattern: coordinating compensation across a sequence of steps that partially succeeded

The saga pattern, first described by Garcia-Molina and Salem, was built specifically to handle what ACID transactions can't: long-running, distributed processes where holding a single lock across every step isn't realistic. Instead of one giant transaction, a saga breaks the operation into a sequence of smaller, locally atomic transactions, and pairs each one with a compensating transaction that reverses it if something downstream fails.

Worth being precise about what "reverse" means here, because it's not a database rollback. A compensating transaction doesn't erase the fact that something happened, it logically undoes the business effect. The ledger still shows the charge and the refund as two separate events. Nothing gets wiped from history.

An order workflow makes this concrete. Reserve inventory pairs with release the reservation. Charge payment pairs with issue a refund. Send confirmation pairs with send a cancellation notice. If the confirmation step fails permanently, the saga executor runs the compensations in reverse order: refund the charge, release the inventory. The customer ends up seeing a failed order, not a charge with no explanation attached.

Here's the part that's easy to miss: the compensating transactions need to be idempotent too. If the orchestrator crashes after running "refund charge X" but before recording that it ran, "refund charge X" has to be safe to call a second time. It needs to check, see the charge already refunded, and return success without moving any money a second time. Idempotency keys on compensation calls aren't a nice-to-have, they're the whole reason the pattern holds up under real failures.

Sagas trade distributed atomicity for eventual consistency. No distributed locks, which is the whole appeal, but the system has to tolerate a window where intermediate state is visible to the outside world, an order that looks "paid" for a few seconds before confirmation lands. As with step classification earlier, the design effort for compensation logic concentrates where the consequences are real: funds moving, notifications going out, authoritative state changing. Steps that are naturally idempotent and leave no external trace usually don't need a compensating action at all.

Applying these patterns in an agent runtime: what the harness must own versus what each tool must own

None of this works if it's left to each individual tool to figure out on its own. There's a layering question underneath all of it: what does the agent runtime, the harness, need to own centrally, and what can safely live inside each tool's own implementation?

The runtime layer is the natural home for workflow-level concerns: tracking what ran, what didn't, generating and checking idempotency keys before a tool call goes out, and driving compensation when something downstream fails. That's coordination work, and it belongs above the level of any single tool, because no individual tool can see the whole sequence it's part of.

Each tool, in turn, needs to own its own side of the contract: making its own operation genuinely idempotent where that's possible, and honoring the deduplication key it's handed rather than treating it as decoration. A payment tool that ignores the idempotency key it's given is just as broken as a runtime that never generates one in the first place. Both halves have to hold for the whole thing to actually work, and a system that gets one right without the other is still one bad retry away from charging somebody twice.