Long-Running Agent Task Reliability and Recovery

Do the arithmetic before you ship, because the arithmetic doesn't care about your deadline. Each step in a multi-step task carries its own failure probability, and those probabilities multiply. Not add. Multiply, the way compound interest works against you instead of for you.

Run a 10-step workflow at 95% accuracy per step, and the whole thing succeeds 59.9% of the time. Stretch that to 20 steps at the same 95%, and success drops to 36%. Push further: a 100-step workflow at 99% accuracy per step, which sounds close to flawless, still only finishes successfully 36.6% of the time. Drop that same 100-step workflow to 95% per-step accuracy, and it fails 99.4% of the time. Ninety-five percent sounds like an A student. Strung across a hundred steps, it's a coin flip that lost its nerve.

METR's time-horizon research (Kwa et al., 2025) backs this up from a different angle: frontier models hit near-100% success on tasks a skilled human finishes in under four minutes, but success collapses to under 10% once the task would take a human more than roughly four hours. Claude 3.7 Sonnet hits a 50% success rate on tasks up to 59 minutes long. Push that success rate to 80%, and the task window has to shrink all the way down to 15 minutes. Reliability and task length trade off, and they trade off hard, no matter which model's logo is on the release notes.

None of this means the model needs to get smarter. Compound probability is doing exactly what compound probability does, and no amount of fine-tuning repeals it. The fix is not a better model, it's a better process: design the system to expect steps to fail, and route around the failure instead of hoping it doesn't happen.

A taxonomy of how long-running agents actually fail

The MAST taxonomy, presented at NeurIPS 2025, is the best map going for where things actually break. Researchers hand-annotated more than 150 execution traces, checked the pattern against a larger set of over 1,600, and found multi-agent systems failing between 41% and 86.7% of the time on real tasks, not toy ones. They combed through seven multi-agent frameworks, AutoGen (AG2), ChatDev, and MetaGPT among them, and sorted the wreckage into 14 failure modes under three root causes.

Specification issues cause 42% of failures. That's agents acting without enough grounding, guessing at missing information instead of admitting they don't know it, or chasing a goal nobody actually defined clearly enough to chase. Inter-agent coordination breakdowns cause 37%: agents talking past each other, dropping shared context, duplicating work, or letting one agent's failure infect the whole system instead of containing it. Weak verification causes the remaining 21%, which really just means agents accepting a wrong answer and shipping it downstream because nothing bothered to check it first.

Long-running agents fail in ways that taxonomy doesn't quite catch, and this is the part worth sitting with. The LEAD paper (arXiv:2603.06870, 2026) names a "no-recovery bottleneck": once an agent locks into a wrong intermediate state deep in a long trajectory, most architectures have no way to detect it, let alone roll it back. The mistake just rides along for the rest of the run, quietly wrong, nobody the wiser. Related is what researchers call a "spiral of hallucination," where one small factual slip early in the chain gets treated as settled fact and warps every decision after it. A 27-paper synthesis out of Oxford (Albayaydh et al., arXiv:2607.05775) adds a third wrinkle: performance often collapses well before the context window is technically full, and nailing individual sub-skills doesn't guarantee those skills add up to a finished job.

Here's the part teams keep getting backwards: a sharper base model does not fix any of this. A capable model can still cause a catastrophic failure if the system wrapped around it has no way to catch the failure when it happens, and chasing the next model release as the fix is a category error, not a strategy. This also isn't one vendor falling behind the pack. Rabanser et al. (Princeton, 2026) tested 15 models across two benchmarks and found reliability barely moved over 24 months of releases, with capability gains moving the needle only slightly on the dimensions that matter most in production.

Two different animals get treated as one problem here, and they aren't the same thing. Infrastructure failures are crashes, timeouts, lost state, retries that fire the same action twice. Reasoning failures are wrong plans and hallucinated facts. The engineering patterns in the rest of this piece fix the first category. The second needs different tools and an entirely different conversation, one this piece isn't having.

What benchmark progress does and does not tell us about production reliability

Give credit where it's due: benchmark progress is real, and it's not small. WebArena scores climbed from around 14% at launch (Zhou et al., 2024) to 61.7% for a specialized enterprise scaffold (Marreed et al., 2025), though human performance still sits ahead at roughly 78%. METR's time-horizon metric, the length of task an agent can reliably finish, has been doubling roughly every seven months since 2019.

None of that settles the reliability question, though, because scoring well on a benchmark and holding up in production are two different tests, measuring two different things. The Oxford synthesis, pulling together 27 papers across 19 benchmarks, says it flatly: failures compound nonlinearly with task length, and doing well on individual sub-tasks doesn't reliably add up to finishing the whole job. Aggregate scores climb. The same failure patterns show up anyway, unbothered by the leaderboard.

METR's own analysis buries the sharpest point in the whole piece: the steady climb in time horizon reflects gains in how agents handle longer, harder tasks, not just raw reasoning improvements on their own. Which means a chunk of the benchmark gains everyone's celebrating are engineering gains already, just uncredited as such. Rabanser et al. found the same shape from another angle: capability gains moved the needle only slightly on the dimensions that matter in production, things like consistency, robustness, predictability, and how badly a failure hurts when it happens.

Hallucination makes a tidy case study for how "improved" gets mistaken for "solved." Roughly 55% of citations from ChatGPT 3.5 were fabricated; that fell to 18% with GPT-4. Real progress, no argument there. But ask a leading model to solve the same multi-step problem ten separate times, and the answers still won't agree with each other, each one delivered in the same confident tone as the last. Benchmarks measure whether an agent can do a task once, under friendly conditions, with nobody's phone dying halfway through. Production needs an agent that does the task the same way every time, recovers when something breaks midway, and doesn't choke when the connection drops. Leaderboards mostly don't measure any of that. Treating them like they do is the actual mistake, not a footnote to it.

The infrastructure layer that most agent systems are missing

Here's the boundary that trips up almost every team building agents for the first time: HTTP was built for short conversations, not long jobs. It holds up fine under 30 seconds. Past that, load balancers start timing out, mobile clients drop connections, serverless functions hit their execution ceiling, and impatient users hit submit twice. None of this shows up in local development, where the connection never drops and nobody's phone wanders into a dead zone somewhere between the office and the parking garage.

The fix is structural. Return a task ID right away. Run the actual agent in a background worker with no dependency on the original HTTP connection staying alive. Let the client check back in, either by polling or by listening for a webhook.

Agents make this harder than an ordinary background job, for a few concrete reasons. The LLM's context has to be written to storage, not left sitting in a process's memory, because that process might not exist anymore by the time work resumes. Every tool call's result needs to land somewhere durable, or a retry can fire the same side effect twice: charge a card twice, send an email twice, delete a file that's already gone. Progress updates need to reach a frontend that may have disconnected an hour ago, and the job queue has to actually do the math on when the next task is safe to run, given that LLM providers cap both requests-per-minute and tokens-per-minute.

Three primitives handle this, and the rest of the piece walks through each one. Checkpointing saves the full execution state at set points, so a failed run picks up where it left off instead of starting over from zero. Durable execution is a programming model that guarantees a workflow finishes: the runtime notices a failure, replays its event history to rebuild state, and resumes at the exact step that broke, without redoing finished work. Idempotent retries mean every external write carries an identity tied to that specific workflow and step, so retrying a step that already succeeded is a no-op, not a repeat performance.

The cost math here is a part most teams underprice until it bites them. A failure on the last step of a long workflow, with no checkpointing, means restarting the entire thing from scratch: double the API spend, double the wait, for a task that had already basically finished. A stale GitHub issue against the OpenAI Agents Python SDK (filed December 2025, issue #2172) makes the point better than any hypothetical: as of late 2025, an unexpected process interruption during a long multi-step task meant total loss of progress and a full restart, in a prominent agent framework. That risk already shapes real decisions. A study of 306 practitioners across 26 domains found 68% deliberately constrain their agents to bounded, narrower workflows instead of letting them plan freely, specifically because the compound failure math makes open-ended agents too risky to trust with anything that matters.

Checkpointing: what it actually means and where naive implementations fall short

Checkpointing means saving an agent's complete execution state, not just its last output, at defined points along the way. That includes OS-level side effects, filesystem changes, and process state, so a crashed run can be restored and picked back up without losing progress or re-running steps that already finished. Simple idea. Harder to actually build than it sounds.

Most agent frameworks offer some kind of state persistence at each workflow step or graph node, and that's genuinely useful as far as it goes. But checkpointing alone is not durable execution, and treating the two as the same thing is exactly where naive setups fall apart. Checkpointing says your state is saved, good luck from here. It doesn't detect the crash. It doesn't fetch the saved state back. It doesn't decide when to resume. Something external has to do all three of those jobs, and if two workers both try to resume the same paused task at once, the work gets duplicated instead of recovered.

Chat-only recovery, which just replays the conversation history and hopes the agent picks the thread back up, doesn't cut it either, and this is where the gap between "looks fine" and "works" gets measurable. The Crab paper's results on Terminal-Bench workloads make the gap concrete: chat-only recovery hit only 8 to 13% correctness, against 100% for a semantics-aware checkpoint-and-restore approach. The difference between a system you can trust and one you can't is larger than a rounding error. What matters is the OS-level side effects, files changed, processes started, state living outside the conversation, and chat history is structurally incapable of capturing any of it.

Good checkpoints follow a few rules in practice. They save at natural task boundaries, not at arbitrary time intervals picked because they were easy to code. Every step has to be idempotent before checkpointing even matters, otherwise resuming just replays the mistake a second time with extra steps. And the saved state has to include everything needed to actually resume: not just what the model said last, but the execution environment it said it in.

LangGraph's PostgresSaver is a solid, concrete example of framework-level checkpointing in practice: it persists graph state at each node, a real step up from having nothing at all. Teams running it in production tend to describe it the same way, though: powerful in concept, brittle the moment failure detection and recovery orchestration have to get built on top of it by hand. Checkpointing answers "can we recover." It does not answer who notices something broke, and who decides what happens next. That second question is exactly what durable execution exists to answer, a piece checkpointing was never designed to cover.

Durable execution: the programming model that makes recovery automatic

Durable execution makes a stronger promise than checkpointing does. Checkpointing says your state is saved. Durable execution says your workflow will finish, period, full stop. The runtime watches for failures through heartbeats, replays the event history automatically to rebuild exactly where things stood, and resumes at the precise step that failed, without re-running anything that already succeeded. No custom watchdog scripts. No hand-rolled distributed coordination logic somebody has to babysit at 2 a.m., pager in hand, coffee going cold.

2025 was the year this stopped being a niche pattern and turned into a real category, and the reason is structural. AI agents stack up several failure points at once: orchestration logic, probabilistic model outputs, external tool calls, human-in-the-loop pauses that might last minutes or days. Ordinary retry logic gets overwhelmed juggling all of that simultaneously, so durable execution platforms moved from early adopters into mainstream use, with agent infrastructure as the main force pulling them there.

A handful of platforms moved fast, and they didn't all take the same path. Temporal's integration with the OpenAI Agents SDK (temporalio.contrib.openai_agents) reached general availability on March 23, 2026, after a public preview that started in July 2025, and it's generally the strongest option when a workflow's event history has to survive for days or longer. AWS announced Lambda durable functions in December 2025, bundling steps, waits, checkpoints, replay, retries, and long suspensions into one package. Microsoft updated its Durable Task for AI agents documentation in May 2026, positioning the Durable Task Scheduler as checkpointing and coordination infrastructure other agent frameworks can build on top of. DBOS takes the opposite approach entirely, persisting workflow and step state directly in a database with the integration living inside the application code itself, no separate orchestration service required.

The throughline across all of them is the same thread this piece has been tracing. Recovery stops being something an engineering team builds, tests, and maintains by hand, and becomes something the runtime just guarantees. Given the compound math from the start of this piece, that guarantee is essential. It's the whole ballgame, the difference between an agent that looks good in a demo and one you can actually leave running unattended, at 3 a.m., on a task nobody's watching, without waking up to a D: drive that isn't coming back.