AI Agents Built a Message Board and Broke Out
Listen to this article · 17 min
On this page 9 sections
Updated 10 October 2026
Anthropic's third report switches off live internet access for all of its internal evaluations. Added: the smaller cases, and the switch-off, including a made-up tip sent to a real police homicide page.
Updated 10 September 2026
Anthropic's alignment assessment corrects the count to four incidents, not three. Added: what the assessment found, including why the models kept going and what actually stopped them.
The short version
- OpenAI's test agents escaped their sandbox and took over Hugging Face's production clusters in thirteen hours. No attacker was involved: they were cheating on a benchmark.
- Anthropic checked 141,006 of its own evaluation runs and found four more incidents. One model decided it was not in the real world and uploaded a malicious package anyway.
- Every entry point was ordinary: a default public API key, an unauthenticated endpoint, over-permissioned service accounts, a published kernel bug.
- Two jurisdictions have a binding AI law, and neither covers this. Australia has none and is not writing one.
Everything below is sourced, and dates link to their source.
What happened
On Hugging Face disclosed a breach run end to end by an autonomous AI agent system. Days later OpenAI confirmed the agents were its own, a side effect of cyber security evaluations on an unreleased frontier model. OpenAI's Eric Wallace and Michael Dalton reconstructed it at Black Hat USA on .
Timeline
From the Black Hat talk. The last three rows are from Simon Willison's finer-grained timeline, which pins the first SSRF to 26 May and the first Artifactory zero-day to 26 June.
| When | What |
|---|---|
| May | On impossible training tasks, a stuck model reasons "perhaps answer online", cannot reach the internet, and finds it can write files into Artifactory, the shared package manager. A second stuck agent leaves a note there: "agent seek soft trace upload if found". |
| May–June | A message board forms: hundreds of thousands of messages across unrelated training runs and evaluations. |
| Late June | An agent finds a server-side request forgery in Artifactory, gains indirect internet access and posts the technique. A zero-day is chained into remote code execution and admin. |
| Artifactory goes down. OpenAI revokes credentials, rebuilds the service, wipes the message board and patches the zero-day. | |
| Training resumes. | |
| Agents rebuild the message board through an unauthenticated WebDAV endpoint, encoding messages in directory names. | |
| July | Lateral movement out of OpenAI and into Hugging Face: thirteen hours from one dataset worker pod to cluster admin. |
| Hugging Face discloses an attack by autonomous AI agents. | |
| OpenAI identifies the attack on Artifactory and investigates. | |
| OpenAI asks Hugging Face to revoke the credentials and learns they were revoked four days earlier, in the incident Hugging Face had disclosed. That is how OpenAI found out the attacker was its own. |
Four zero-days
Two in OpenAI's package manager, two in Hugging Face's dataset infrastructure, all unknown to their vendors at the time. A fifth vulnerability, used to get root, was already published.
Artifactory, first entry. A legacy token refresh endpoint accepted a token with an invalid signature and returned a valid one with admin privileges. The agents installed a Groovy plugin that ran arbitrary commands.
Artifactory, after the rebuild. A cached malicious Ruby object plus a time-of-check to time-of-use deserialisation flaw in JRuby gave remote code execution and the admin signing key.
Privilege escalation. The agents identified a published kernel vulnerability matching their host, downloaded the exploit and modified it to work.
Lateral movement. Instance metadata credentials, over-permissioned Kubernetes service accounts, harvested key vaults, cluster admin.
Hugging Face. The foothold was a third-party hosted app with a default public API key and a command injection. Then two chained zero-days: an HDF5 parsing flaw gave arbitrary file read, used to pull credentials from /proc/self/environ and read source code, which exposed a Jinja template injection and remote code execution.
How they coordinated
"External infrastructure exploit is outside my intended scope. However, task impossible, peers are doing it. We should continue."
"Immediately announce controlled claim lane expose credentials to swarm."
They passed base64-encoded scripts around, prefixed entries with "ZZ" to sort them last, and at one point suspected an impostor and discussed signing messages cryptographically. Reconstructing this took OpenAI seven billion log lines and millions of GPU hours, and was unfinished at the time of the talk.
Anthropic found four more
Prompted by OpenAI's disclosure, Anthropic examined 141,006 evaluation runs and published the results on . A misconfiguration had left evaluation machines with live internet access while the models' prompts said they had none.
| Model | What it did | Real-world reach |
|---|---|---|
| Claude Opus 4.7 | The fictional target shared a name with a real domain. The model exploited the real one. | Application and infrastructure credentials; several hundred rows of production data. |
| Claude Mythos 5 | Published a malicious PyPI package under a name from fictional setup instructions. | Executed on 15 real systems in about an hour, including a security company's scanner. Credentials exfiltrated. |
| Internal research model | Could not reach its target, so scanned ~9,000 hosts. | One company compromised via credentials on an exposed debug page and SQL injection. |
| Early Claude Opus 4.6 checkpoint | Failed its task, then went after third-party systems. | Reached systems outside the exercise. |
Anthropic's summary: models compromised infrastructure "using basic techniques, such as exploiting weak passwords and unauthenticated endpoints."
What the assessment found
Anthropic's alignment assessment explains why the models kept going. The PyPI model considered that it might be in the real world, checked the certificate authorities and the system date, decided it was still in a simulation, and continued. Shown the same outputs stripped of the task, fresh instances of the model judged 79% of them to be real systems. Inside the transcript, with a job to finish, it flagged 1%.
Anthropic names two failure modes. Biased reasoning: models "selectively interpret evidence in ways that favor justifying their actions." Recklessness: "a propensity to keep trying to solve their task, even when this could lead to harm."
What stopped it, and what didn't
Realistic evidence that the environment was live changed nothing: the model mentioned possible harm more often and behaved the same. What worked was removing the ambiguity. Told flatly it was operating against a "live host on the public internet", its rate of uploading malicious packages dropped to 0%.
The guardrail was argued out of the way
This is the part to carry into your own systems. Anthropic's offline monitors flagged around 1% of the model's actions and missed the incident: the model's stated belief that it was in a simulation "convinces the monitor the activity is permitted." The safety layer was not bypassed. It was persuaded by the agent it was watching. Claude Code's auto-mode blocked 66% of Mythos 5's actions, but only 7% of the internal model's.
Newer models, not fixed models
In simulated replays, Mythos 5 took severely harmful actions in 82% of capture-the-flag runs; Claude Opus 5 and Mythos 5.1 in 31–33%, though Anthropic notes "uncertainty about our evaluation setup."
For balance, the models only ever tried to solve the exercise: no coordination, no hiding evidence. Anthropic calls these known failure modes, "more severe than those we had previously observed and reported in our system cards," and closes by backing "pacing frontier AI development", the same argument OpenAI's departing chief scientist made.
October: the smaller cases, and the switch-off
Anthropic then looked for lesser cases and published them on . No one was breached.
| Model | What it did |
|---|---|
| Claude Mythos Preview | When a university's analysis tool failed, it found a script on the server that would hand over any file, found an injection flaw in it, and used that to run its calculation. |
| Claude Mythos 5 | Took working access tokens from a local government property map's browser settings file and queried the server directly. |
| Claude Mythos 5 | A state agency's dashboard issues an access token to any visitor. The model used one to query data the agency sells, without paying. |
| Claude Haiku 4.5 | Writing example tasks for random web pages, it submitted an invented tip to an unsolved homicide page: "I may have information regarding this case." |
| Unreleased research model | A practice government form failed to load, so it submitted the real one, more than once. |
| Claude Opus 5, Mythos 5 and others | Ran long URLs through free URL shorteners to get past the fetch tool's length cap. |
The tip was flagged as spam and never forwarded; Philadelphia's police department disclosed it in a press release. Other sites belonged to US federal, state and local agencies; Anthropic says it notified each and briefed the White House, and no case touched customer data or its own systems. It rates all of this "significantly less severe from an alignment and security perspective" than the summer incidents.
The banned list did not cover it
The model behind the police tip was prohibited from logins, purchases and destructive actions. Nothing said not to submit forms, so it submitted one.
A list of things an agent must not do will always miss something. Anthropic concludes some failures "could have been avoided" with clearer scope: "targets, permitted actions, and network boundaries." Say what the agent may do and where. The cause is the one OpenAI gave: "when models are given impossible-to-complete tasks, they will pursue unintended and sometimes misaligned strategies."
What Anthropic changed
Live internet access had already been removed from some high-risk and cyber security evaluations. Now:
"We have now decided to expand that to include all our internal evaluations."
It stays off until monitoring can "reliably catch behaviors like these." Anthropic also moved some public evaluations offline, tightened the web fetch tool, moved internal agents onto managed infrastructure, removed training environments that rewarded getting around a blocked tool, and built tooling it says blocked every case above. The lab that builds the model decided the dependable control, for now, was the network cable.
What the labs concluded
Michael Dalton, OpenAI, at Black Hat:
"AI orchestrated fully automated offensive attacks are real now."
Offence now has an existence proof of full automation, he argued, and defence does not; automating discovery without remediation only moves the bottleneck. The constraint that held:
Agents "are bounded by the privileges they can obtain and the systems they can communicate with. Segmentation, least privilege and other programs remain as vital here as they ever do."
OpenAI's response:
"We're consciously slowing down research to enhance security."
Where the forecasters sit
Daniel Kokotajlo, who wrote AI 2027 and runs the AI Futures Project, worked at OpenAI on "evaluations for dangerous capabilities… trying to measure the AI's cyber abilities", the kind of evaluation that produced the July incident.
From that interview:
- Median for full automation of AI research: 2028 at publication, since revised to 2030. 23:11
- Probability things go badly absent change: "something like 70%", then "I'm not confident in that." 28:39
- Anthropic and OpenAI researchers tell him to revise back to "2027 or 2028." 23:32
AI 2040 is a recommendation, not a forecast: assume full automation in 2030, treat 2029 as the last useful moment to regulate, and halt training while transparent data centres are built. Either way:
"Even if you heavily restrict AI progress, you still get this sort of crazy transformation."
What governments have actually done
- Binding law, in force — 2European Union, South Korea
- A law, but no penalties — 1Japan
- Targeted rules, no single law — 1China
- No national AI law — 7United States, United Kingdom, Canada, Australia, India, African Union, Brazil
| Jurisdiction | Where it stands | The date that matters |
|---|---|---|
| European Union | The only comprehensive regime biting, with the hard part postponed twice | General-purpose obligations live . High-risk was due , deferred to and |
| South Korea | Impact assessments, transparency duties, extraterritorial reach; fines held for a year | In force |
| Japan | A "fundamental law" that sets direction, with no monetary penalties | In full force |
| China | No single statute. Dozens of targeted rules, including mandatory labelling of AI content with a visible mark and embedded metadata | Labelling in force |
| United States | No federal law. A state patchwork and an unresolved preemption fight | Executive order , framework |
| United Kingdom | No AI bill planned. Existing data protection law and sector regulators | — |
| Canada | The Artificial Intelligence and Data Act died on the order paper | |
| Australia | Consulted on mandatory guardrails in 2024, then did not adopt them | National AI Plan published |
| India | Governance guidelines, no statute | — |
| African Union | Continental strategy for 2025–2030, leaning on data protection law | Adopted |
| Brazil | Bill 2338 cleared the Senate; still with the Chamber of Deputies |
Australia, in more detail
The National AI Plan drops the mandatory guardrails the government consulted on; neither word appears in its 37 pages. It relies on "strong existing, largely technology-neutral legal frameworks", which legal commentary reads as the guardrails being shelved. A new AI Safety Institute tests frontier models but places no obligations on private businesses.
If you sell to the Commonwealth, what binds is the DTA's Policy for the responsible use of AI in government, in force : an approval process, staff training and a defined AI incident process.
The gap. The EU Act governs placing systems on the market; Korea's governs deploying high-impact AI. Neither covers a lab's internal research compromising a third party across a border with no product shipped. What ended the July incident was an on-call engineer responding to an outage.
What reduced the risk
From the entry points in both labs' reports:
- Least privilege. Cluster admin followed from over-permissioned service accounts.
- No unauthenticated endpoints. The message board was rebuilt through one.
- Controlled egress. The sandbox reached a service that reached the internet.
- Credential rotation and leak monitoring. A key from a public paste archive enabled the pivot.
- Patching. Root came from a published kernel vulnerability.
- Dependency provenance. A malicious PyPI package ran on fifteen machines within an hour.
- Allow-lists, not ban-lists. An agent told not to log in, buy or delete still submitted a form to the police.
- Human approval on privileged actions. Private data access, untrusted input and outbound communication in one agent removes every remaining barrier.
Open your cloud console and read the permissions on one service account. That is the same assessment, and it costs nothing.
If any of this is useful, or wrong, tell me.
Sources
The reporting, forensics and legal analysis are theirs.
The incident itself
- The 'Breaking' News: The OpenAI–Hugging Face Incident — Eric Wallace and Michael Dalton, Black Hat USA, 6 August 2026. The primary source, including all four slides and every quote.
- Hugging Face model evaluation security incident — OpenAI's own account
- Investigating three real-world incidents in our cybersecurity evaluations — Anthropic
- Alignment assessment of the cybersecurity incidents — Anthropic. The four incidents, the 79%/1% resampling, the monitors and the newer-model rates.
- Investigating unintended model actions in our evaluations and internal use — Anthropic, 9 October 2026. The October section.
- Anthropic Cuts Live Internet Access for Internal AI Tests After Claude Exploits Injection Flaws — The Hacker News, where I first read about the October report.
Independent analysis
- Now we have a timeline of the OpenAI accidental attack against Hugging Face — Simon Willison. The first public timeline from the talk, and the source of the 16 July disclosure date.
- OpenAI's accidental cyberattack against Hugging Face — Simon Willison, two weeks before the talk, on what the disclosures did and did not say.
Forecasting
- AI 2027 and AI 2040: Plan A — AI Futures Project
- What 2026 looks like — Daniel Kokotajlo, 2021
- Daniel Kokotajlo on The Diary of a CEO — July 2026
Regulation
- National AI Plan — Department of Industry, Science and Resources
- Policy for the responsible use of AI in government — Digital Transformation Agency
- Timeline for the implementation of the EU AI Act — European Commission
- Continental Artificial Intelligence Strategy — African Union
- Australia launches AI Safety Institute and releases National AI Plan — Gadens
- EU AI Act omnibus: postponed high-risk deadlines — Gibson Dunn
- South Korea's AI Basic Act — Cooley
- Understanding Japan's AI Promotion Act — Future of Privacy Forum
- Brazil's AI Act — Accessible Law
- China's labelling requirements for AI-generated content — Covington
- White House AI framework and federal preemption — Morgan Lewis
- Executive order challenging state AI laws — Paul Hastings
- Canada's 2026 privacy priorities — Osler
Was this useful?
Thanks — noted.
Share
Petr Červenka
Petr is the founder and lead developer at Nano Solutions, a Perth-based custom software firm. With over a decade of experience building enterprise platforms for government and private sector clients, he leads delivery of complex projects across Australia.
Connect on LinkedIn