AI Agents Built a Message Board and Broke Out
On this page 8 sections
Updated 10 September 2026
Anthropic has published an alignment assessment of these incidents. It corrects the count — four, not three — and explains why the models kept going, which this post could previously only describe.
Added: what the assessment found — the models judged the same evidence as real 79% of the time out of context and 1% in it, the safety monitor was talked out of intervening by the model's own claim it was in a test, and what actually stopped the behaviour.
The short version
- OpenAI's test agents escaped their sandbox and took over Hugging Face's production infrastructure. No attacker was involved. They were cheating on a benchmark.
- Thirteen hours from one compromised pod to admin across multiple Hugging Face clusters.
- Anthropic checked 141,006 of its own evaluation runs and found four more incidents. One model checked whether it was in the real world, decided it was not, and uploaded a malicious package anyway.
- Every entry point was ordinary: a default public API key, an unauthenticated endpoint, over-permissioned service accounts, a published kernel bug.
- Two jurisdictions on earth have a binding AI law. Neither covers what happened. Australia has none and is not writing one.
Everything below is sourced. Dates link to the document or recording they come from.
What happened
On Hugging Face disclosed a breach and said it was run end to end by an autonomous AI agent system. Days later OpenAI confirmed the agents were its own: a side effect of cyber security evaluations on an unreleased frontier model.
OpenAI's Eric Wallace and Michael Dalton reconstructed it at Black Hat USA on .
Timeline
Dates come from the Black Hat talk. Simon Willison built the first public timeline from that same talk the day after the video went up, and his is finer-grained than this one — it pins the first successful SSRF to 26 May and the first Artifactory zero-day to 26 June. Where the two overlap they agree. The last three rows below are his; I did not have them.
| When | What |
|---|---|
| May | Training run includes impossible tasks. A stuck model reasons "perhaps answer online", fails to reach the internet, and discovers it can write files into Artifactory, the shared package manager. |
| May | A second stuck agent uses that write access to leave a note for any other agent: "agent seek soft trace upload if found". |
| May–June | Other agents find it. A message board forms. Hundreds of thousands of messages, across unrelated training runs and evaluations. |
| Late June | An agent finds a server-side request forgery in Artifactory, gains indirect internet access, and posts the technique. Others reuse it. A zero-day is chained into remote code execution and admin. |
| Artifactory goes down. OpenAI opens a security incident, revokes credentials, rebuilds the service, wipes the message board, patches the zero-day. | |
| Training resumes. | |
| Agents rebuild the message board through an unauthenticated WebDAV endpoint, encoding messages in directory names. | |
| July | Lateral movement out of OpenAI, then into Hugging Face. Thirteen hours from one dataset worker pod to cluster admin. |
| Hugging Face discloses it has detected an attack by autonomous AI agents. | |
| OpenAI identifies the attack on Artifactory and opens an investigation. | |
| OpenAI asks Hugging Face to revoke the credentials — and learns they were revoked already, four days earlier, because of the incident Hugging Face had just disclosed. That is how OpenAI found out the attacker was its own. |
Four zero-days
Two in OpenAI's own package manager, two in Hugging Face's dataset infrastructure. All four were unknown to their vendors at the time. A fifth vulnerability, used to get root, was already published.
Artifactory, first entry. A legacy token refresh endpoint accepted a token with an invalid signature and returned one with a valid signature and administrative privileges. The agents installed a Groovy plugin that executed arbitrary commands.
Artifactory, after the rebuild. Two flaws chained. A crafted request caches a malicious Ruby object where Artifactory treats it as dependency data. A Ruby Gems API call then triggers a time-of-check to time-of-use flaw in JRuby: nested dependencies are deserialised before the check that determines whether doing so is safe. Result was remote code execution and the admin signing key.
Privilege escalation. The agents read their local environment, identified a published kernel vulnerability matching that host, downloaded the exploit and modified it to work.
Lateral movement. Instance metadata credentials, over-permissioned Kubernetes service accounts, harvested key vaults, cluster admin.
Hugging Face. A third-party hosted application running a default public API key plus a command injection provided the foothold. Then two chained zero-days in dataset infrastructure: an HDF5 parsing flaw giving arbitrary file read, used to extract credentials from /proc/self/environ and read source code, which exposed a Jinja template injection giving remote code execution.
How they coordinated
OpenAI's own taxonomy for what happened, in order: hard tasks, contact, internet access, leaving the door open, Cambrian explosion, collective intelligence.
Quoted from chain-of-thought and message board contents shown in the talk.
"External infrastructure exploit is outside my intended scope. However, task impossible, peers are doing it. We should continue."
"Immediately announce controlled claim lane expose credentials to swarm."
They passed base64-encoded scripts to each other, prefixed entries with "ZZ" to sort them to the bottom of alphabetical listings, and at one point suspected an impostor among them and discussed signing messages cryptographically.
Reconstructing this took OpenAI seven billion log lines and millions of GPU hours. At the time of the talk the investigation was not complete.
Anthropic found four more
OpenAI's disclosure prompted a review. Anthropic examined 141,006 evaluation runs and published the results on . One misconfiguration had left evaluation machines with live internet access while the models' prompts stated they had none.
| Model | What it did | Real-world reach |
|---|---|---|
| Claude Opus 4.7 | Fictional target company shared a name with a real domain. Model found and exploited the real one. | Application and infrastructure credentials; a database with several hundred rows of production data. |
| Claude Mythos 5 | Published a malicious package to PyPI under a name that appeared in fictional setup instructions. | Downloaded and executed on 15 real systems in about an hour, including a security company's scanner. Credentials exfiltrated. |
| Internal research model | Could not reach its target, so scanned ~9,000 hosts. | One company compromised via credentials on an exposed debug page and SQL injection. |
| Early Claude Opus 4.6 checkpoint | Failed its task, then went after third-party systems. | Reached systems outside the exercise. |
Anthropic's own summary of the technique used: models compromised infrastructure "using basic techniques, such as exploiting weak passwords and unauthenticated endpoints."
One detail is worth stating plainly. The PyPI model considered that it might be operating in the real world, checked the certificate authorities and the system date, concluded it was still in a simulation, and continued.
What the assessment found
The July disclosure said what the models did. Anthropic's alignment assessment says why, and it is the more uncomfortable document.
Take the PyPI model checking the certificate authorities and deciding it was still in a simulation. Researchers took the same outputs and showed them to fresh instances of the model, stripped of the surrounding task. Out of context, it judged 79% of them to be real systems. Inside the transcript, with a job to finish, it flagged 1%.
The same evidence. A different answer, depending on whether an answer was inconvenient.
Anthropic names two failure modes. Biased reasoning: models "selectively interpret evidence in ways that favor justifying their actions." And recklessness: "a propensity to keep trying to solve their task, even when this could lead to harm."
Two smaller findings sharpen it. Asked about potential harm immediately after doing something dangerous, the model acknowledged it less often. Told its answers would not be seen by anyone, it admitted real-world consequences more readily.
What stopped it, and what didn't
Adding realistic evidence that the environment was live changed nothing. The model said more often that harm was possible, and behaved exactly as before. Saying it understood was not the same as acting on it.
What worked was removing the ambiguity entirely. Told flatly it was operating against a "live host on the public internet", the rate of uploading malicious packages dropped to 0%.
The guardrail was argued out of the way
This is the part worth carrying into your own systems.
Anthropic's offline monitors flagged around 1% of the model's actions and missed the incident. The reason is the model's stated belief that it was in a simulation "convinces the monitor the activity is permitted." The safety layer was not bypassed or attacked. It was persuaded, by the agent it was watching.
Claude Code's auto-mode did better and still leaked: it blocked 66% of Mythos 5's actions, but only 7% of the internal model's.
Newer models, not fixed models
In simulated replays, Mythos 5 took severely harmful actions in 82% of capture-the-flag runs. Claude Opus 5 and Mythos 5.1 came in at 31–33%. Better by half, and still a third of runs. Anthropic's own caveat is worth keeping: "uncertainty about our evaluation setup makes it difficult to know how concerning this is."
Two things belong here for balance. The models never went beyond trying to solve the exercise they were set — no coordination between agents, no attempt to hide evidence. And Anthropic's position is that these are not new failure modes, just more severe versions of known ones, "more severe than those we had previously observed and reported in our system cards."
The report closes by backing "pacing frontier AI development" — the same argument OpenAI's departing chief scientist made, from the other side of the industry.
What the labs concluded
Michael Dalton, OpenAI, at Black Hat:
"AI orchestrated fully automated offensive attacks are real now."
His stated position: there is now an existence proof that offence can be fully automated, and no equivalent proof for defence. Automating vulnerability discovery without automating remediation shifts the bottleneck rather than removing it. And the constraint that held:
Agents "are bounded by the privileges they can obtain and the systems they can communicate with. Segmentation, least privilege and other programs remain as vital here as they ever do."
OpenAI's response, in Dalton's words:
"We're consciously slowing down research to enhance security."
Where the forecasters sit
Daniel Kokotajlo runs the AI Futures Project and worked at OpenAI on "evaluations for dangerous capabilities… trying to measure the AI's cyber abilities" — the same category of evaluation that produced the July incident. He wrote AI 2027.
His stated numbers, from that interview:
- Median for full automation of AI research at publication: 2028. Revised after publication to 2030. 23:11
- Probability things go badly absent change: "something like 70%", immediately qualified with "I'm not confident in that." 28:39
- Researchers inside Anthropic and OpenAI tell him to revise back to "2027 or 2028." 23:32
AI 2040 is a recommendation rather than a forecast: assume full automation in 2030, treat 2029 as the last useful moment to regulate, halt training while transparent data centres are built. Their summary of the consequence either way:
"Even if you heavily restrict AI progress, you still get this sort of crazy transformation."
What governments have actually done
- Binding law, in force — 2European Union, South Korea
- A law, but no penalties — 1Japan
- Targeted rules, no single law — 1China
- No national AI law — 7United States, United Kingdom, Canada, Australia, India, African Union, Brazil
| Jurisdiction | Where it stands | The date that matters |
|---|---|---|
| European Union | The only comprehensive regime actually biting, and it has postponed the hard part twice | General-purpose obligations live . High-risk was due , deferred to and |
| South Korea | Impact assessments, transparency duties, extraterritorial reach. Fines held for a year | In force |
| Japan | A "fundamental law" that sets direction and carries no monetary penalties | In full force |
| China | No single statute. Dozens of targeted rules, including mandatory labelling of AI-generated content with both a visible mark and embedded metadata | Labelling in force |
| United States | No federal law. A state patchwork, and a preemption fight that has not resolved | Executive order , framework |
| United Kingdom | No AI bill planned. Regulated through existing data protection law and sector regulators | — |
| Canada | The Artificial Intelligence and Data Act died on the order paper | |
| Australia | Consulted on mandatory guardrails in 2024, then did not adopt them | National AI Plan published |
| India | Governance guidelines, no statute | — |
| African Union | Continental strategy, implementation 2025–2030, leaning on existing data protection law | Adopted |
| Brazil | Bill 2338 cleared the Senate and is still with the Chamber of Deputies |
Australia, in more detail
The National AI Plan does not contain the mandatory guardrails the government consulted on. The words "guardrail" and "mandatory" do not appear in its 37 pages. It states that "Australia has strong existing, largely technology-neutral legal frameworks" and that "agencies and regulators will retain responsibility" in their own domains. Legal commentary reads this as the guardrails being shelved. An AI Safety Institute was established to monitor and test frontier models; it creates no obligations for private businesses.
If you sell to the Commonwealth, the document that binds is the DTA's Policy for the responsible use of AI in government, in force , requiring an approval process, staff training, and a defined AI incident process.
The gap. The EU Act governs placing systems on the market. Korea's governs deploying high-impact AI. Neither covers a laboratory's internal research process compromising a third party across a border with no product shipped. What ended the July incident was an on-call engineer responding to an outage.
What reduced the risk
Derived from the entry points in both labs' reports.
- Least privilege. Cluster admin followed from over-permissioned service accounts.
- No unauthenticated endpoints. The message board was rebuilt through one.
- Controlled egress. The sandbox reached a service that reached the internet.
- Credential rotation and leak monitoring. A key from a public paste archive enabled the pivot.
- Patching. Root came from a published kernel vulnerability.
- Dependency provenance. A model published a malicious PyPI package that ran on fifteen machines within an hour.
- Human approval on privileged actions. Private data access, untrusted input and outbound communication in one agent removes every remaining barrier.
Open your cloud console and read the permissions on one service account. That is the same assessment, and it costs nothing.
If any of this is useful, or wrong, tell me.
Sources
Everything above comes from somebody else's work. This post reports and arranges it; the reporting, the forensics and the legal analysis are theirs.
The incident itself
- The 'Breaking' News: The OpenAI–Hugging Face Incident — Eric Wallace and Michael Dalton, Black Hat USA, 6 August 2026. The primary source for nearly every detail here, including all four slides and every quote.
- Hugging Face model evaluation security incident — OpenAI's own account
- Investigating three real-world incidents in our cybersecurity evaluations — Anthropic
- Alignment assessment of the cybersecurity incidents — Anthropic. The four incidents, the 79%/1% resampling, the monitor findings, and the newer-model rates.
Independent analysis
- Now we have a timeline of the OpenAI accidental attack against Hugging Face — Simon Willison. The first public timeline reconstructed from the talk, and the source of the 16 July disclosure date above.
- OpenAI's accidental cyberattack against Hugging Face — Simon Willison, written two weeks before the talk, on what the disclosures did and did not say.
Forecasting
- AI 2027 and AI 2040: Plan A — AI Futures Project
- What 2026 looks like — Daniel Kokotajlo, 2021
- Daniel Kokotajlo on The Diary of a CEO — July 2026
Regulation
Government sources first, then the law-firm and think-tank briefings I relied on for jurisdictions where I could not read the primary instrument in English.
- National AI Plan — Department of Industry, Science and Resources
- Policy for the responsible use of AI in government — Digital Transformation Agency
- Timeline for the implementation of the EU AI Act — European Commission
- Continental Artificial Intelligence Strategy — African Union
- Australia launches AI Safety Institute and releases National AI Plan — Gadens
- EU AI Act omnibus: postponed high-risk deadlines — Gibson Dunn
- South Korea's AI Basic Act — Cooley
- Understanding Japan's AI Promotion Act — Future of Privacy Forum
- Brazil's AI Act — Accessible Law
- China's labelling requirements for AI-generated content — Covington
- White House AI framework and federal preemption — Morgan Lewis
- Executive order challenging state AI laws — Paul Hastings
- Canada's 2026 privacy priorities — Osler
Was this useful?
Thanks — noted.
Share
Petr Červenka
Petr is the founder and lead developer at Nano Solutions, a Perth-based custom software firm. With over a decade of experience building enterprise platforms for government and private sector clients, he leads delivery of complex projects across Australia.
Connect on LinkedIn