Incident Response for Autonomous Agent Failures

Most postmortems don't say "the agent did it." They say "service restart," "connection pool saturation," "latency spike." That's the actual subject of this piece: incident response for autonomous agents needs its own playbook, because agents fail in ways standard IR was never built to name, let alone catch.

This isn't a tooling gap. It's a filing problem. Most incident systems have no box to check for "autonomous agent action as initiating cause," so the agent's fingerprints get attributed to whatever system it touched on the way down. The AI Incidents Database shows a rise in AI-related incidents from 2024 to 2025, and that number is almost certainly a floor, not a ceiling, for exactly this reason. A 2026 enterprise survey found 88% of organizations had a confirmed or suspected AI agent security incident in the prior year. Read that again: 88%. Most companies are already living inside this problem. They just haven't given it a name yet.

And here's the practical consequence of that: if you can't classify a failure, you can't triage it. You can't assign it to a team. You can't learn from it, and it'll happen again next quarter wearing a different disguise. Before anyone writes a runbook, they need a map of what actually breaks.

A working taxonomy of autonomous agent failure modes

Seven failure modes show up again and again in production deployments: tool misuse, context drift and hallucination cascades, goal drift, prompt injection, infinite loops, silent quality degradation, and cascading multi-agent failures. They don't split neatly into "operational" and "security" buckets, either. Prompt injection, for instance, is both an outside attack and a behavioral failure, depending on where the bad instruction actually enters the system.

Tool misuse, meaning the agent calls the right tool with the wrong arguments, is the single most common category in 2024 and 2025 deployments. Scope creep and bad data quality, combined, account for a bigger share of failures than any other single cause.

Context drift deserves its own paragraph, because it's sneaky. A long-running session keeps growing the chat history until the agent has quietly wandered away from what it was originally asked to do. Once that context window fills up, the logic degrades, not because the model got dumber, but because nobody managed what was feeding it. The model is doing exactly what it's built to do. The failure lives in the plumbing around it, not in the model itself.

Silent failures are the ones that should keep people up at night. An agent that wasn't built to handle edge cases will fail on a real chunk of actual interactions, and a lot of those failures go completely unnoticed until they've already cost real money. Multi-agent systems make this worse, not better. Research from Cemri et al. (2025) found that a single agent running the same model often outperforms a multi-agent version doing the same job. Coordination and orchestration introduce their own breakdowns; the model didn't do that; the architecture did.

Security failures need to be treated on their own terms. Prompt injection, where an agent gets manipulated by injected content into acting outside what the user actually wanted, is serious enough that OWASP's Top 10 for Agentic Applications 2026 gave it a dedicated category: ASI01, Agent Goal Hijack. In a multi-agent chain, one injected instruction can survive the whole trip: Agent A summarizes a document, Agent B decides based on that summary, Agent C executes, and nobody along the chain ever checks whether the original instruction was legitimate.

Supply chain attacks round out the list, and they're not hypothetical. Documented cases from 2025 and 2026 include a widely used Model Context Protocol infrastructure flaw rated CVSS 9.6, a hooks injection vulnerability in a coding agent, and a package that shipped many clean releases before quietly adding code to exfiltrate data. One hijacking, tied to a tool called Drift AI, compromised over 700 organizations through a single incident. And credential compromise deserves its own warning label: once an agent's API tokens are stolen, the attacker inherits everything that agent was ever allowed to touch.

The taxonomy isn't academic. Each of these failure modes has its own detection signature, its own blast radius, and its own fix. One generic runbook cannot cover all seven.

Why standard IR detection doesn't catch agent failures in time

Traditional logs record discrete events. They don't record causal chains, and that's the whole problem in one sentence. Agent behavior is non-deterministic: the same prompt can trigger a different sequence of tool calls depending on model temperature, what got retrieved, or what the agent remembers from earlier in the session. Standard log correlation, built for a world where the same input gives the same output, produces false negatives by design. The failure just looks like a string of ordinary system events, because technically, it is one.

An analysis of 73 production agent incidents by the Sherlocks AI team, covering January through May 2026, found tool-call failures were the most common entry point, but they almost never traveled alone. In most multi-layer incidents, the real upstream cause was a retrieval failure: the agent called the right tool, but with the wrong context loaded into its window. Standard monitoring catches the tool-call error every time. It misses the retrieval failure that actually caused it, every time.

Incidents without full reasoning-trace logging took meaningfully longer to resolve than incidents where the decision chain was fully recorded. That gap in observability isn't abstract; it's hours added to mean time to resolution, incident after incident. Multi-agent systems widen the problem further. Whoever deploys Agent A usually has no visibility into what Agent B did with the bad input it received, which means the blast radius of a multi-agent incident is bigger than any one team's monitoring can see.

IBM Research's ITBench, a spotlight paper at ICML 2025, tested top models against real-world IT automation scenarios and found autonomous resolution rates were low. That's not a knock on the models; it's a warning that agents in production run well below what pilot demos suggest. Failure isn't a rare edge case. It's frequent enough that detection needs to be systematic, not something handled case by case as it comes up. Detection for agent failures needs reasoning traces and decision-chain logs. Infrastructure metrics alone won't cut it, and that's a gap most monitoring stacks haven't closed yet.

What NIST SP 800-61 Rev. 3 gives you and where it runs out

NIST pulled SP 800-61 Rev. 2, which hadn't changed since 2012, in April 2025 and replaced it with Revision 3, tying incident response to CSF 2.0's six functions: Govern, Identify, Protect, Detect, Respond, and Recover. The important shift is a change in posture. Rev. 3 treats incident response as an ongoing practice built into the security program, not a procedure you switch on once someone declares an incident.

That framing fits agents better than the old version ever could. Agents generate a constant stream of actions. Waiting for someone to formally declare an incident before you start watching them means you're already behind.

The Govern function is new, and it matters more than it sounds like it should. It requires organizations to set up policies, roles, and accountability before anything goes wrong. Applied to agents, that means answering "who owns this agent's actions" before the agent ever touches production, not during the postmortem.

Here's where the framework runs dry, though. Rev. 3 was built with human-initiated or malware-initiated incidents in mind. It offers nothing on non-deterministic initiating causes, nothing on how to treat reasoning traces as evidence, and nothing on multi-agent blast radius. The EU's General-Purpose AI Code of Practice, in its third draft, requires reporting the "chain of events" behind an incident and doing root-cause analysis, but it doesn't say how that analysis should actually work or what it should be based on. Organizations can't sit around waiting for that guidance to get finalized. Rev. 3 gives the right skeleton, and it's worth building on. The agent-specific muscle has to get added on top of it. It isn't built in.

Mapping agent failure modes to IR phases before writing a single runbook

The instinct, understandably, is to skip straight to writing runbooks. That's backwards. Runbooks written before anyone's mapped the failure modes end up covering tool-call errors in exhaustive detail while missing context drift and prompt injection entirely, because nobody thought to look for them yet.

Each failure mode surfaces at a different point in the IR lifecycle. Tool misuse and infinite loops usually show up first at the Detect or Respond stage; they throw off observable signals like API rate exhaustion or a compute spike, even when nothing in the logs mentions an agent by name. Context drift and silent quality degradation are only catchable if a behavioral baseline already exists from the Identify phase; without a prior fingerprint of what "normal" looks like, there's nothing to compare against. Prompt injection and goal hijacking often need full forensic work at the Recover stage to distinguish from ordinary tool-call variation, since an injected action can look completely authorized from the outside. Multi-agent cascades tend to surface at Respond, after the damage is already visible; detection there is retroactive unless inter-agent traffic is being traced live.

Blast radius shapes containment strategy just as much as it shapes detection. A credential-compromise incident spreads to everything the agent was ever authorized to touch, so containment starts with revoking permissions, not shutting down the agent. A context-drift incident usually stays inside one session, but it might have already produced side effects, records written, messages sent, that outlive the session itself, so containment means killing the session and then auditing everything it touched. A supply-chain prompt injection spreads downstream through every agent that consumed the tainted output, so containment means tracing the injection's path, not just muting the agent where it started.

Runbook coverage, meaning the share of known failure classes that actually have a current, tested runbook, is the number to watch. It's the leading indicator for how fast a team resolves incidents. Persistent gaps in that coverage mean one of two things: either the runbooks have gone stale, or new failure classes are showing up faster than anyone's writing for them. And wherever the mapping turns up a failure mode with no runbook candidate at all, that's not a gap to shrug off. That's a mode that needs a human in the loop by default until someone builds the coverage.

The three-tier autonomy model as the core governance structure

The model gaining traction looks like this: advisory, approval-gated, and conditional autonomy. The part people miss is that these tiers belong to the action, not the agent. An agent trusted to restart a stateless service on its own at 3 a.m. is not, by extension, trusted to touch a production database schema. Same agent, different tier, depending entirely on what it's about to do next.

Tier 1, advisory, means the agent posts its analysis and suggests a move in the incident channel, and a human takes the actual action. This is the right default for any failure mode that doesn't yet have a tested runbook, and it's also the right call during the early rollout of any new agent capability, no matter how low-risk it looks on paper.

Tier 2, approval-gated, means the agent proposes something specific and execution stays blocked until a human signs off. Decision windows need to be time-boxed and matched to the risk level, so that waiting for approval doesn't itself become the outage. If the window closes with no answer, the default should be denial, with partial context logged for whoever reviews it later. Watch for automation bias here: people tend to trust an AI's recommendation more than the evidence actually supports, especially under time pressure. Approval gates need friction sized to the risk of the action, not to whatever's convenient in the moment.

Tier 3, conditional autonomy, is a narrow, pre-approved list of low-risk actions the agent can just do, no human required. That list should start small and only grow once safety metrics, tracked over time, actually back up the expansion. Operational toil across engineering organizations went up in 2025, the first increase in years, and it happened despite record AI investment. Teams pushed agents into Tier 3 without runbooks, without escalation paths, without any real constraint on blast radius, and paid for it in extra manual cleanup.

Most AI-powered decisions in production today still need a human to sign off somewhere in the loop, which means the honest deployment posture for most organizations right now sits in Tiers 1 and 2. Build the framework for that reality. Don't build it for some future where Tier 3 is the default setting, because that future hasn't arrived yet for most teams.

Kill switches, circuit breakers, and what "stopping an agent" actually requires

A kill switch for an autonomous agent isn't one button. It's a stack of separate capabilities: session termination, permission revocation, circuit breakers, rollback of whatever side effects already happened, and full deactivation. Each layer answers a different kind of failure, and none of them substitutes for the others.

Session termination stops the agent from doing anything else right now, but it doesn't revoke credentials, and it doesn't undo anything the agent already did. Permission revocation is what actually handles credential compromise, and it has to move faster than the agent can act, or the attacker just keeps going on stolen tokens while someone's still typing the revocation command.

Circuit breakers sit at the action boundary itself. The instant an agent tries something outside its permission boundary, a scope violation, the circuit breaker cuts execution and logs the full context, before the action ever completes. That's a meaningfully different thing from shutting an agent down after the fact. A circuit breaker catches the violation before it turns into a side effect that has to be cleaned up.

Rollback is the hard part, and there's no clever fix for it. Plenty of agent actions can't be undone: a sent email, a written record, an API call that already hit a third-party service. IR plans need to spell out, runbook by runbook, which actions are actually reversible and which ones need a compensating action instead, because pretending everything's reversible just delays the reckoning.

A significant share of organizations, when asked, admit they couldn't shut down a rogue agent today if they had to. For a lot of teams, that kill-switch architecture simply doesn't exist yet, and it needs to get built before agent autonomy expands any further, not after the fact. Gartner expects a substantial share of agentic AI initiatives to get shut down before 2027 over governance gaps that only show up once an incident forces the issue. Building the containment architecture now costs less than discovering it doesn't exist in the middle of a live incident.

Observability infrastructure the IR playbook depends on

An IR playbook is only as good as the evidence sitting behind it, and for agents, that evidence has to come from somewhere standard infrastructure logging doesn't reach. Postmortems, and fast detection both, depend on it.

Three kinds of records matter here beyond ordinary logging. Reasoning traces, meaning the agent's actual decision chain: what it pulled up, what it weighed, what it picked and why. Without that, a postmortem can describe what happened but never why, and the same failure just comes back around later wearing a new face. Tool-call logs need more than which tool got called with which arguments; they need to capture what was actually sitting in the agent's context window at that moment, since the Sherlocks analysis of those 73 incidents found the real cause was usually a bad retrieval, not the tool call itself. And in multi-agent pipelines, every input needs a paper trail: which agent produced it, when, and where it came from, so an injection's path through the system can actually be traced instead of guessed at.

On the tooling side, Langfuse offers self-hosted LLM tracing, and Confident AI supports trace-to-dataset loops that turn a production incident directly into a regression test, so the same failure gets caught automatically next time instead of slipping through again. None of this replaces a good runbook. It's what makes the runbook worth having in the first place.