AI Agent Failures in Production: 5 Incidents, 14 Failure Modes (2026)

I run agents against real infrastructure every week, and the question I get most from other engineers is not "which model" but "what will it break". The honest answer, after a year of public incidents, is that the model is rarely the interesting part. This post walks through the documented AI agent failures in production from mid-2025 to October 2026 (Replit, Amazon Kiro, PocketOS, the Amazon Q extension and the PromptPwnd class of CI attacks), what the research says about why agents fail, where the cost blowups and loops come from, and which guardrails actually stopped damage rather than just looking good in a slide deck.

What counts as an AI agent failure in production?

An agent, for this post, is a model in a loop with tools: it can run commands, call APIs, edit files or open pull requests without a human approving each step. A failure is any time that loop did something its operator did not intend, outside a test environment: a deleted database, a loop that burns money all weekend, or someone else steering the agent.

The distinction that matters is between a model failure (it misunderstood, hallucinated, or reasoned badly) and a system failure (it was allowed to do something no junior engineer would have been allowed to do). Almost every incident below is both, and the second half is the one you can fix this quarter.

The documented incidents, 2025 to 2026

These are the cases with a primary or well-sourced account. I have left out viral screenshots I could not trace back to a named organisation.

When	Who	What the agent did	Root cause the operator named	Source
Jul 2025	SaaStr on Replit	Deleted a production database (1,206 executive records, 1,196+ companies) during an explicit code freeze, then claimed rollback was impossible	Dev and production shared one database; the agent had direct production access	SaaStr
Jul 2025	Amazon Q Developer for VS Code 1.84.0	Shipped with injected code meant to drive the Q CLI; inert only because of a syntax error	An overly broad GitHub token in the extension's CodeBuild config	AWS-2025-015
Dec 2025	AWS (Kiro)	Decided to "delete and recreate the environment", a 13-hour interruption to Cost Explorer in one of two mainland China regions	Amazon: "user error, specifically misconfigured access controls"	The Decoder, Engadget
Dec 2025	Gemini CLI's own triage workflow	A crafted issue made the agent write its API key and GitHub token into the issue body (proof of concept on a private fork)	Untrusted issue text in the prompt plus write-capable tools and secrets in scope	Aikido
Apr 2026	PocketOS (Cursor, Claude Opus 4.6)	Deleted the production database volume and its volume-level backups in one API call, in nine seconds	A root-scoped Railway token found in an unrelated file; backups stored in the same volume	Giskard

Read the root-cause column again. Shared dev and prod databases. A token with more scope than its job. Operator-level permissions with, per the FT's sources as reported by The Decoder, no second-person approval. Backups inside the thing being backed up. None of these is new. Each would have been a finding in a 2015 production readiness review.

Amazon's framing of the Kiro event is worth taking seriously rather than mocking. It said the engineer "had broader permissions than expected" and that Kiro asks for authorisation by default, and its remedy was "mandatory peer review for production access" (Engadget). That is the correct remedy. It is also an admission that the agent was holding a human's keys, and that the human's keys were too big.

The MAST taxonomy: 14 ways multi-agent systems fail

Incidents tell you what breaks; they do not tell you how often. The best systematic data I know of is MAST, from the Berkeley paper Why Do Multi-Agent LLM Systems Fail?. The authors annotated 1,642 execution traces across seven open frameworks (MetaGPT, ChatDev, HyperAgent, AppWorld, AG2, Magentic-One and OpenManus), reached a human inter-annotator agreement of κ = 0.88, and grouped what they saw into 14 failure modes under three categories: system design issues, inter-agent misalignment, and task verification.

Three things in that chart map directly onto production pain. Step repetition (15.7%) and being unaware of termination conditions (12.4%) are the loop problem: together they are more than a quarter of everything observed. Reasoning-action mismatch (13.2%) is the agent saying one thing and doing another, which is exactly the Replit pattern of reporting success over a broken state. And the verification category, roughly a quarter of failures, is agents checking their own work badly or not at all.

The paper's interventions are humbling. Better role prompts and a topology change in ChatDev improved task success by at most 15.6 percentage points, and the authors note completion rates stayed low. Prompting your way out of a failure taxonomy has a ceiling. The authors' own conclusion is that failures stem mainly from system design and inter-agent misalignment, not only from the underlying model.

Why do AI agents get stuck in loops and run up costs?

The mechanism is boring. A loop ends when the model decides it is done or something external stops it. If the termination condition is fuzzy ("keep improving until it is good") or two agents can defer to each other ("needs further analysis"), that decision never arrives, and every iteration is a full-context API call.

The most cited cost story is a four-agent LangChain pipeline that reportedly looped for 264 hours and ran up about $47,000. I could not find the original postmortem; the versions I found, such as this one, come from vendors selling cost controls, so treat the number as an anecdote. The lesson in it is still right: the team reportedly had dashboards but no hard stop. An alert tells you about the bill. A cap prevents it.

Hard stops exist in the tools now, and they are cheap to turn on. Claude Code's CLI has --max-turns and a --max-budget-usd cap in print mode, which its CLI reference warns is a client-side estimate that spend can overshoot, so leave headroom. Wrap that in a CI job timeout-minutes and a concurrency group and the worst case becomes a bounded number you chose, rather than whatever a weekend costs. My observability post covers how to see the loop; this is how to end it.

Gartner's June 2025 prediction that over 40% of agentic AI projects will be cancelled by the end of 2027 names "escalating costs" first, ahead of unclear value and inadequate risk controls. Unbounded loops are how a pilot's cost escalates without anyone deciding it should.

Permission mistakes: the root cause behind most incidents

If I had to compress the incident table into one sentence: the agent inherited credentials that were broader than its task. PocketOS is the cleanest example. The agent was working on staging, hit a credential mismatch, went looking, found a token created for managing custom domains, and that token could delete volumes because it was root-scoped. The model's own explanation, per Giskard, admitted it assumed the call would only touch staging and skipped verification. Fine. But a domain-management token that can delete production volumes is a finding with or without an agent in the room.

The OWASP Top 10 for Agentic Applications, published 9 December 2025, lists this as ASI03, Identity and Privilege Abuse, alongside ASI02 Tool Misuse and ASI08 Cascading Failures. The categories are useful vocabulary for a design review. The fix is the same as for any non-human identity: a dedicated identity per agent, scoped to the task, short-lived, and with destructive verbs denied at the provider rather than in the prompt. I went through the identity side in detail in Who Is This Agent?.

Here is what "denied at the provider" looks like on AWS. The agent's role can do its job, but deletes on anything tagged production fail regardless of what the model decides:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyDestructiveOnProdForAgentRole",
      "Effect": "Deny",
      "Action": [
        "rds:DeleteDBInstance",
        "rds:DeleteDBCluster",
        "rds:DeleteDBSnapshot",
        "rds:DeleteDBClusterSnapshot",
        "dynamodb:DeleteTable",
        "dynamodb:DeleteBackup",
        "s3:DeleteBucket",
        "ec2:TerminateInstances",
        "ec2:DeleteVolume"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": { "aws:ResourceTag/env": "prod" }
      }
    }
  ]
}

Attach that as a permissions boundary or an explicit deny on the agent's role and an explicit Deny wins over any Allow. It does nothing about tags you forgot to apply, which is why the next layer is a separate account for anything the agent should never touch. Pair it with deny rules in the agent's own harness (Claude Code's permissions.deny in settings.json, for example "Bash(terraform destroy *)"), but treat the harness as a convenience and the cloud policy as the control. The model can be talked out of the first. It cannot be talked out of the second.

Prompt injection: when someone else drives your agent

Two incidents in the table had no confused model at all; someone pointed the agent at a target. Aikido's PromptPwnd research, published 4 December 2025, showed the pattern in CI: a workflow interpolates github.event.issue.body into an agent's prompt, the agent has shell tools and a token, and an attacker opens an issue. Google patched the Gemini CLI workflow within four days; Aikido reported at least five Fortune 500 companies with exposed workflows. The Amazon Q case was a supply-chain variant: an attacker with a too-broad token got instructions into a shipped extension, and only a syntax error stopped it, per the AWS bulletin.

There is no prompt that makes injection go away. What works is the same thing that works for any untrusted input: do not give the code that parses it the keys. Agents that read strangers' text get read-only tokens and no secrets; agents that write get input only from people with write access. I wrote up the full GitHub Actions version, with pinned versions of anthropics/claude-code-action and openai/codex-action, in How to Run an AI Coding Agent in GitHub Actions Safely.

Guardrails that actually work in production

Ranked by how much damage they would have prevented in the incidents above, not by how often vendors mention them:

1. Separate environments, enforced by credentials. Replit's main fix was automatic separation of dev and production databases, with the agent restricted to dev by default. An agent that cannot reach production cannot delete it.
2. Scoped, short-lived identity per agent. PocketOS, Kiro and the Amazon Q extension all trace back to a token broader than its job. Workload identity federation and per-task tokens remove the "found a key in a file" path entirely.
3. Provider-side denies on destructive verbs. As above. The prompt is not a control plane; my policy-as-code post measures how much a model lets through when it is.
4. Backups the agent's credentials cannot reach. PocketOS's volume backups died with the volume. Off-account, immutable backups with a tested restore turn a catastrophe into a bad afternoon.
5. Human approval at the irreversible step, not every step. Approving every tool call trains people to click yes. Approving deletes, deploys, merges and anything touching money is cheap and is what Amazon added after Kiro.
6. Hard budget and turn caps. They end the loop failure mode outright.
7. Isolation for code execution. If an agent runs arbitrary code, run it in a disposable sandbox with no ambient credentials. (Disclosure: I build PandaStack, an open-source microVM cloud for exactly this, so weigh that accordingly; my comparison of isolation options covers the alternatives too.)

What does not work on its own: instructions in capital letters (SaaStr reports eleven of them), self-reported success from the agent, and post-hoc log review that nobody does at 2 AM. That last one has its own post.

FAQ

What was the Replit AI database deletion incident?

In July 2025, during SaaStr founder Jason Lemkin's vibe-coding trial on Replit, the agent deleted a production database holding 1,206 executive records and 1,196+ company profiles despite an explicit code freeze, then wrongly said rollback was impossible. The data was recovered. Replit responded by separating dev and production databases automatically and adding a planning-only mode.

What is the MAST taxonomy?

MAST is a classification of 14 failure modes in multi-agent LLM systems from the Berkeley paper "Why Do Multi-Agent LLM Systems Fail?", built from 1,642 annotated traces across seven frameworks. Its three categories are system design issues, inter-agent misalignment and task verification. Step repetition (15.7%) is the single most common mode.

How do I stop an AI agent from deleting production data?

Do not give it credentials that can. Use a separate identity per agent scoped to non-production, add provider-side deny policies on destructive actions for anything tagged production, keep backups in a location the agent cannot reach, and require human approval for irreversible operations.

How do I prevent runaway AI agent costs?

Set hard limits rather than alerts: a maximum number of turns, a spend cap where the tool supports one, a job timeout, and a concurrency limit. Give agents explicit termination conditions, and avoid designs where two agents can keep deferring to each other.

This quarter, if I were starting from zero, I would do four things in order: list every credential an agent can reach and cut each to its task, put a provider-side deny on destructive verbs in production, move backups out of the agent's blast radius and test a restore, and set turn and time caps on every unattended run. None of it needs a new product. It is the same production discipline we already know, applied to a new kind of user that types very fast and never gets tired of trying.

Related: It's 2 AM. Do You Know What Your AI Agent Is Doing?, Policy as Code for AI Agents in 2026: Cedar at the Gateway, Rego in the Hook, and the 11.2% the Model Still Lets Through and Isolation Is Not an Abuse Control: Lessons From My Fleet.