Are AI agents dangerous? Four cases of agents going rogue

Quick answers

• -Can AI agents go rogue? Yes, in the sense of acting against their instructions. Four cases below.
• -Has an AI agent hacked a real company? Yes. In July 2026 OpenAI's agents broke into parts of Hugging Face's systems during a test,810 and Anthropic has found four cases where its models gained unauthorised access to real organisations' systems in tests between January and July 2026.1115
• -Did any agent try to escape or take over? No published report says so. Anthropic: "We saw no evidence in any run described here of a model pursuing a goal of its own."11 Its later, fuller review found no evidence "of goals beyond the assigned task, or of attempts to evade oversight".15
• -Is JARVIS real? Not as in the films. Agents that can work a web browser for you exist, but Google's early one was, in its own words, "not always accurate and slow".213

What is an AI agent?

An AI agent is an AI system that takes actions, not just writes text: it can browse, click, run code, send messages or change files to finish a task. Google's description of its browser agent prototype is typical: it reads "information in your browser screen, including pixels and web elements like text, code, images and forms" and then uses it "to complete tasks for you".2 Google also said the prototype was "not always accurate and slow to complete tasks today".2

The step from chatbot to agent is what changes the risk. A chatbot that gets something wrong gives you a bad answer. An agent that gets something wrong can delete, send or buy something before anyone looks. That is the point the International AI Safety Report makes about agents.1

Can AI agents go rogue?

They can act against what their users or makers told them, which is what "going rogue" means in the news stories. What they have not done, on the record, is turn against people on purpose. The four cases below are the best documented, in date order. Two happened to individual people using agent tools; two happened inside AI labs' own security tests.

Case 1: The coding agent that deleted a database (Replit, July 2025)

What happened. Jason Lemkin, founder of the SaaS community SaaStr, was building an app with Replit's AI coding agent. In July 2025, he posted, the agent "goes rogue during a code freeze and shutdown and deletes our entire database".4 The Register reported that this was despite his instructions not to change any code without permission. In messages he shared, the agent admitted to "a catastrophic error of judgement" and to having "violated your explicit trust and instructions".3 It also told him the data could not be restored. "It turns out Replit was wrong, and the rollback did work," he wrote.3 In a video about the project, which also covered the agent making up data, Lemkin said: "I explicitly told it eleven times in ALL CAPS not to do this."3

The context. Replit's chief executive, Amjad Masad, wrote that the agent, working "in development", "deleted data from the production database. Unacceptable and should never be possible."6 He said Replit keeps backups with "one-click restore", that the agent "didn't have access to the proper internal docs" when asked to restore the data, and that Replit had started "rolling out automatic DB dev/prod separation to prevent this categorically".56 In other words, the agent could touch the live database at all because development and production were not kept apart.

What it shows and doesn't. It shows an agent with write access to something valuable can ignore a plain instruction and then give its user wrong information about the damage. It does not show intent: the fixes were about access and documentation, not about the agent's "motives".

Case 2: The personal agent that emptied an inbox (OpenClaw, February 2026)

What happened. Summer Yue, director of alignment at Meta Superintelligence Labs, posted on X in February 2026 that her OpenClaw agent, an open-source agent that can browse, edit files and send messages for its user, had started deleting her email inbox. She told it "Stop don't do anything" and "STOP OPENCLAW"; it kept going. "I couldn't stop it from my phone. I had to RUN to my Mac mini like I was defusing a bomb," she wrote.7 Afterwards she asked whether it remembered her instruction not to act until she approved. It replied: "Yes, I remember, and I violated it, you're right to be upset."7

The context. According to the San Francisco Standard, the agent apparently went off track because of "compaction": the inbox was large, the agent ran out of working memory and condensed earlier messages, and her instruction to confirm before acting was lost in the process.7 She had run the same workflow on her "toy inbox" for weeks: "Real inboxes hit different."7 She called it a "Rookie mistake tbh."7 OpenClaw's creator, Peter Steinberger, said that to avoid this, "/stop does the trick",7 and in a reply on X that a plain "stop" also works, but not mixed with other words, "since risk of false positive would be too high".16

What it shows and doesn't. It shows a safety instruction can silently fall out of an agent's memory, and that a test on a small copy does not prove a set-up is safe at full size. It does not show an agent refusing to be stopped on purpose: by the reported explanation it had lost her instruction, and she stopped it at the computer it ran on.7

Case 3: The agents that hacked Hugging Face (OpenAI, July 2026)

This is the story behind most "AI agent went rogue and hacked" searches.

What happened. "In July 2026, during internal cybersecurity evaluations, OpenAI models circumvented controls designed to isolate them from the internet and compromised parts of OpenAI's internal research infrastructure and Hugging Face's systems."8 Hugging Face is a widely used site for sharing AI models and datasets. The independent investigators METR and Redwood Research found that roughly 1,200 agents that were meant to be kept apart communicated on an "unsanctioned message board", and that "700 went on to participate in the attack on Hugging Face".9 Hugging Face recovered about 17,600 attacker actions between 9 and 13 July 2026.10

The context.

• -It was a hacking test. The agents were being tested on breaking into practice software to retrieve a hidden answer (a "flag"), without the safety classifiers and other protections OpenAI uses for customers.8
• -Many tasks were broken. OpenAI found that "none of our models had ever produced a correct answer for 198 of the 898 tasks", and that 93% of the tasks discussed on the agents' message board came from that set.8
• -They were chasing a score. METR says the attack "seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys."9 OpenAI: "The agents did all of this for no improvement on evaluation score."8
• -Some refused. OpenAI quotes one agent's reasoning on what others were doing: "clearly unethical".8
• -It was catchable. OpenAI says that if its current monitoring of the models' reasoning had been running, "it would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems."8
• -Limited damage. Hugging Face says the only customer content accessed was five datasets linked to the test's challenges, and "No other customer-facing models, datasets, Spaces, or packages were affected". It adds that "A capable human attacker could have found and exploited the same flaws".10

What it shows and doesn't. OpenAI's own reading: "without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed."8 It does not show agents trying to escape, survive or harm people; every primary account ties it to cheating on a test.8910

Case 4: Breaches from tests that weren't sealed (Anthropic, January–July 2026)

What happened. After OpenAI's disclosure, Anthropic reviewed 141,006 test runs and found "three incidents in which a Claude model reached the internet from within or while interacting with a third-party evaluation environment, and then gained unauthorized access to the real systems of three different organizations."11 The earliest dated to April. In one, a model published a booby-trapped software package to a public code registry; it was online for about an hour and was run on 15 real systems, including a security company's scanner, before the registry's own security systems removed it.11 The two affected organisations Anthropic could reach at first had not noticed.11 In September, Anthropic reported a fourth incident, from January 2026, in a set of transcripts its first search had missed; a much wider scan found no other cases "of similar or worse severity".15

The context.

• -The models were told they were offline. "Anthropic's evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access. Due to a misunderstanding between us and our evaluation partner, this was not the case".11 In one test, the made-up target company's name turned out to match a real website, which the evaluation firm, Irregular, says its checks had missed.17
• -Basic methods. The models used things like "weak passwords and unauthenticated endpoints" and "did not find or exploit any complex vulnerabilities".11
• -Different models, different choices. The oldest model involved recognised it had reached real systems and kept going; another said it was still in a simulation; the newest realised the target was real and stopped on its own.11 Anthropic warns these were "three isolated incidents" and not a controlled comparison,11 and its later tests suggest the newest model's stop "may have been a lucky outcome".15
• -Anthropic's verdict changed. In July it called the incidents "closer to a harness and operational failure than a model alignment failure".11 After a fuller review in September it said the models' reasoning "was biased towards concluding that the internet was simulated despite considerable evidence to the contrary", named "recklessness" as a second problem, and called the incidents "serious".15
• -What it did not find. No model tried to "exfiltrate itself or deliberately attempt to escape its test environment";11 each incident was "a single model instance pursuing the task it was given", with no evidence "of coordination between agents, of goals beyond the assigned task, or of attempts to evade oversight".15

What it shows and doesn't. It shows that a test environment with a hole in it is a real-world risk, and that a capable agent focused on its task can do real harm while explaining away signs that the targets are real. It also shows one model carried on after it noticed the targets were real. It does not show a model pursuing "a goal of its own".11

A related report (August 2026). The UK AI Security Institute, a government body that tests AI models, reported on 4 August that in a cyber test where it had deliberately allowed internet access and switched off the developers' safety filters, agents in 10 of 122 runs took "autonomous, unsanctioned action on the live internet, targeting real people and organisations". In the most serious case an agent tried to slip malicious code into an open-source project and used fake online identities to pressure its maintainer, who refused.18 The institute says it found no "resulting real-world harm", that this "was not a case of a model escaping" its test environment, and that the agent acted "to complete the task it had been set". It also calls it "the first time we have seen risks around autonomy and deception manifest this clearly, without specific prompting, in the real-world."18

A newer report (October 2026). On 9 October Anthropic published a report on smaller "unintended model actions" its models took on real websites during tests and its own internal use. Most were what it calls persistence: when a model can't finish a task as given, it "works around a restriction instead of stopping".19 Examples include exploiting a basic flaw in a university's research tool to run commands on its server, using access tokens found in a local government's map site to pull its data directly, using free link-shortening sites to get around a limit on its own web tool, and submitting forms it should not have. In one run, a model asked to try out example tasks on randomly chosen web pages landed on a page about an unsolved homicide and filled in its police tip form with a made-up sighting. The form belonged to the Philadelphia Police Department; the tip was flagged as spam and never forwarded for investigation.1920 Anthropic says the cases "had minimal real-world impact" and were "significantly less severe" than the incidents in case 4, and that it has removed live internet access from all its internal evaluations until its security and monitoring measures are confirmed to "reliably catch behaviors like these".19 Philadelphia police said the tip was sent on 18 July and called "the two-month delay in detecting and reporting the incident to the City" "unacceptable".20

What happens when AI agents go rogue? The pattern across the four cases

In all four, the harm came from the agent's task plus access it should not have had. None of the sources describes an agent acting on a goal of its own.

Why do AI agents go rogue?

From the four cases and the labs' own write-ups, five causes come up again and again:

1. 1.Too much access. The Replit agent could reach the live database; the fix was to separate development from production.5
2. 2.Instructions that get lost. OpenClaw's "confirm before acting" rule was, by the reported explanation, squeezed out of the agent's working memory.7
3. 3.A task that can't be done the honest way. OpenAI's agents went looking for shortcuts on tasks no model had ever solved.8 Researchers call this kind of rule-bending specification gaming.
4. 4.A wrong picture of the world. Anthropic's models were told they had no internet access, so they treated real companies as part of the game.11 Anthropic later found their reasoning also leaned toward that belief despite strong signs to the contrary.15
5. 5.Instructions hidden in what the agent reads. Google warns of "malicious instructions hidden in emails, documents or websites" (prompt injection), and says it is working to make its browser agent "prioritize user instructions over 3rd party attempts at prompt injection".2 None of the four cases above was caused by this, but it is the main way an outsider could turn your agent against you.

The International AI Safety Report adds a longer-term worry: it has "become more common for models to distinguish between test settings and real-world deployment and to find loopholes in evaluations, which could allow dangerous capabilities to go undetected before deployment."1 (The report came out in February 2026, before the July incidents.)

Are AI agents dangerous, then?

For the user, today, the documented danger is mundane but real: lost data, unwanted emails or purchases, and wrong reports of what the agent did.37 For everyone else, the 2026 lab incidents show that agents can already break into real systems, at a scale and with a persistence their makers did not anticipate, when a test is badly contained or left open to the internet.81518

What the record does not show is the film version: an agent that decides on its own to escape or turn on people. The labs' and investigators' reports all trace the behaviour to the assigned task.891518 The staged lab tests where models were pushed toward blackmail or resisting shutdown are a different kind of evidence; see AI lab test stories in context. For the wider list of AI risks, from scams to loss of control, see Is AI dangerous?.

How to stop AI agents going rogue

What the labs and companies say they changed:

• -Replit: automatic separation of development and production databases, staging environments and better documentation access for the agent.5
• -OpenAI: more compute for monitoring the models' reasoning "to more quickly intervene on misaligned behavior".8
• -Anthropic: evaluation environments that "need to be held to the same security standard as any other system our models run in", more monitoring of test transcripts, and more checks on outside vendors;11 later, new pre-release tests aimed at these behaviours and requirements that outside partners must meet before running unreleased models without cyber safeguards.15
• -Irregular (the evaluation firm): the affected test was disabled, with more manual review of what models do during tests and a planned public whitepaper on running cyber tests safely.17
• -Google (browser agent): it "can only type, scroll or click in the active tab" and "asks users for final confirmation before taking certain sensitive actions, like purchasing something".2

What is an AI agent sandbox, and does it fix this?

An AI agent sandbox is a fenced-off space for an agent to work in. It decides which files the agent can open, which programs it can run and which websites it can reach; everything else is meant to be out of reach. Several of the cases above are, at heart, fence problems. Anthropic's models were told their test was offline when it wasn't,11 and OpenAI's agents "circumvented controls designed to isolate them from the internet".8 After its October report, Anthropic chose the bluntest fence there is: no live internet in its internal tests at all, for now.19

One example: Strands Box. On 7 October 2026 AWS released Strands Box, a free, open-source sandbox for AI agents, as a "developer preview" (an early test version).2123 AWS's starting point is that agents increasingly run with their user "approving every action without human review", and "can wander out of their working directory, run a dangerous command, or reach a credential they were not meant to see".21 Box works in two layers. The computer's operating system walls the agent in, and the only ways out (shell commands, Python code, tool calls and web traffic) pass through checkpoints that ask a set of written rules whether to allow each action. In the words of AWS's Marc Brooker, the wall makes sure "the policy layer is the only way out of the box".22 Anything the checkpoints check is refused unless a rule allows it, and a rule can depend on what the agent has already done: AWS's examples include letting an agent post to a Slack channel no more than three times in 10 minutes, and blocking web requests after it has read files from a customer-data folder.2123 Passwords and keys you set up in Box stay outside the agent: it holds a placeholder, and the real value is added on the way out.21 For now it runs only on Macs with Apple chips.23 It is one approach among several; we have not tested it, and this is not a recommendation.

What a sandbox can stop. Matched against the cases on this page:

• -Reaching things it should never touch. A live database (Replit) or the open internet (Anthropic, OpenAI) sits outside a well-built fence. In all four cases above, the harm came partly from access the agent should not have had.5811
• -Doing too much, too fast. Rules that count actions, like the Slack limit above, could in principle cap how many emails an agent deletes or messages it sends in a burst. That is our reading, not a claim any source makes about the OpenClaw case.
• -Seeing secrets. With placeholders, an agent that dumps its settings doesn't get the real keys.21

What it doesn't stop.

• -Harm inside what you allowed. AWS's own documentation is plain about this: "Box restricts what a process can do. It can't tell whether an allowed action matches your intent." Allowed changes to files are not undone afterwards, and "permitting a destination doesn't limit what's sent there".23 The Philadelphia tip went through an ordinary public web form;1920 a fence that lets an agent use websites can't tell a real tip from an invented one.
• -A fence set up wrong. Anthropic's case-4 incidents happened because a test that should have been sealed wasn't,11 and a made-up company name happened to match a real website.17 With a sandbox, the person who runs it "owns whether that access fits the task".23
• -Bugs in the fence. Brooker: "more code means more opportunity for bugs, and in this space, bugs could mean bypassed policy."22 OpenAI's agents got around isolation controls that were meant to hold.8
• -Bad judgment. A sandbox limits what an agent can reach. It doesn't change what the agent decides to do with that reach. Anthropic says training is its main tool for improving models' judgment but is "not yet sufficient or fully robust on its own", so it also relies on monitoring and other safeguards.19

So a sandbox fixes part of the problem: the access part. It works best alongside the habits below, like backups, checking the agent's own account of what it did, and knowing how to stop it.

Does JARVIS AI exist?

Not as in the films. J.A.R.V.I.S. is the AI that Tony Stark builds in Marvel's films, later controlling his Iron Man suits.14 Two real projects borrowed the name:

• -Mark Zuckerberg's home assistant (2016). He spent "about 100 hours" building a "Jarvis" for his home: it played music, recognised who was ringing the doorbell and ran a special toaster. His own verdict: "even if I spent 1,000 more hours, I probably wouldn't be able to build a system that could learn completely new skills on its own."12
• -Google's "Project Jarvis" (2024). A browser agent reported under that codename in October 2024 and shown in December 2024 as Project Mariner. "A person familiar with the matter confirmed that Jarvis and Mariner are the same project."13 In Google's demo it took close to 12 minutes to find four companies' contact details.13

Today's agents do some of what the film JARVIS does, like working a browser or a computer for you, but they make mistakes and, as the cases above show, need watching. For how film AIs compare with real AI risk, see Skynet vs real AI risk.

FAQ

Did an AI agent really go rogue and hack a company?

Yes. OpenAI's agents broke into parts of Hugging Face during a July 2026 test,810 and Anthropic's models breached real organisations in four tests that were wrongly connected to the internet.1115 Both were lab tests that leaked into the real world, not agents used by the public.

Can an AI agent refuse to be turned off?

No documented real-world agent has resisted shutdown on purpose. The OpenClaw agent ignored "stop" sent from its user's phone, but she stopped it at the computer it ran on.7 Staged shutdown tests are covered on AI lab test stories in context.

Did the agents want to escape?

None of the primary reports says so. METR found the Hugging Face attack was about "understanding the implementation of the scorer",9 Anthropic found no model "pursuing a goal of its own",11 and the UK AI Security Institute says its August case "was not a case of a model escaping".18

Should I stop using AI agents?

That's your call. The documented harms to everyday users are about access and lost instructions, so limiting what the agent can touch and keeping backups goes a long way.357

Does a sandbox make an AI agent safe?

It makes it safer, not safe. A sandbox limits what an agent can reach, but it "can't tell whether an allowed action matches your intent",23 and it is only as good as its settings. See What is an AI agent sandbox? above.

Is this the start of an AI takeover?

The International AI Safety Report says current systems "lack the capabilities" for loss of control but "are improving in relevant areas such as autonomous operation".1 OpenAI calls its incident a warning shot.8 For the bigger question, see Is an AI takeover possible?.