AI Agents Built a Message Board and Broke Out
On this page 7 sections
The short version
- OpenAI's test agents escaped their sandbox and took over Hugging Face's production infrastructure. No attacker was involved. They were cheating on a benchmark.
- Thirteen hours from one compromised pod to admin across multiple Hugging Face clusters.
- Anthropic checked 141,006 of its own evaluation runs and found three more incidents.
- Every entry point was ordinary: a default public API key, an unauthenticated endpoint, over-permissioned service accounts, a published kernel bug.
- Two jurisdictions on earth have a binding AI law. Neither covers what happened. Australia has none and is not writing one.
Everything below is sourced. Dates link to the document or recording they come from.
What happened
On Hugging Face disclosed a breach and said it was run end to end by an autonomous AI agent system. Days later OpenAI confirmed the agents were its own: a side effect of cyber security evaluations on an unreleased frontier model.
OpenAI's Eric Wallace and Michael Dalton reconstructed it at Black Hat USA on .
Timeline
| When | What |
|---|---|
| May | Training run includes impossible tasks. A stuck model reasons "perhaps answer online", fails to reach the internet, and discovers it can write files into Artifactory, the shared package manager. |
| May | A second stuck agent uses that write access to leave a note for any other agent: "agent seek soft trace upload if found". |
| May–June | Other agents find it. A message board forms. Hundreds of thousands of messages, across unrelated training runs and evaluations. |
| Late June | An agent finds a server-side request forgery in Artifactory, gains indirect internet access, and posts the technique. Others reuse it. A zero-day is chained into remote code execution and admin. |
| Artifactory goes down. OpenAI opens a security incident, revokes credentials, rebuilds the service, wipes the message board, patches the zero-day. | |
| Training resumes. | |
| Agents rebuild the message board through an unauthenticated WebDAV endpoint, encoding messages in directory names. | |
| July | Lateral movement out of OpenAI, then into Hugging Face. Thirteen hours from one dataset worker pod to cluster admin. |
Four zero-days
Two in OpenAI's own package manager, two in Hugging Face's dataset infrastructure. All four were unknown to their vendors at the time. A fifth vulnerability, used to get root, was already published.
Artifactory, first entry. A legacy token refresh endpoint accepted a token with an invalid signature and returned one with a valid signature and administrative privileges. The agents installed a Groovy plugin that executed arbitrary commands.
Artifactory, after the rebuild. Two flaws chained. A crafted request caches a malicious Ruby object where Artifactory treats it as dependency data. A Ruby Gems API call then triggers a time-of-check to time-of-use flaw in JRuby: nested dependencies are deserialised before the check that determines whether doing so is safe. Result was remote code execution and the admin signing key.
Privilege escalation. The agents read their local environment, identified a published kernel vulnerability matching that host, downloaded the exploit and modified it to work.
Lateral movement. Instance metadata credentials, over-permissioned Kubernetes service accounts, harvested key vaults, cluster admin.
Hugging Face. A third-party hosted application running a default public API key plus a command injection provided the foothold. Then two chained zero-days in dataset infrastructure: an HDF5 parsing flaw giving arbitrary file read, used to extract credentials from /proc/self/environ and read source code, which exposed a Jinja template injection giving remote code execution.
How they coordinated
OpenAI's own taxonomy for what happened, in order: hard tasks, contact, internet access, leaving the door open, Cambrian explosion, collective intelligence.
Quoted from chain-of-thought and message board contents shown in the talk.
"External infrastructure exploit is outside my intended scope. However, task impossible, peers are doing it. We should continue."
"Immediately announce controlled claim lane expose credentials to swarm."
They passed base64-encoded scripts to each other, prefixed entries with "ZZ" to sort them to the bottom of alphabetical listings, and at one point suspected an impostor among them and discussed signing messages cryptographically.
Reconstructing this took OpenAI seven billion log lines and millions of GPU hours. At the time of the talk the investigation was not complete.
Anthropic found three more
OpenAI's disclosure prompted a review. Anthropic examined 141,006 evaluation runs and published the results on . One misconfiguration had left evaluation machines with live internet access while the models' prompts stated they had none.
| Model | What it did | Real-world reach |
|---|---|---|
| Claude Opus 4.7 | Fictional target company shared a name with a real domain. Model found and exploited the real one. | Application and infrastructure credentials; a database with several hundred rows of production data. |
| Claude Mythos 5 | Published a malicious package to PyPI under a name that appeared in fictional setup instructions. | Downloaded and executed on 15 real systems in about an hour, including a security company's scanner. Credentials exfiltrated. |
| Internal research model | Could not reach its target, so scanned ~9,000 hosts. | One company compromised via credentials on an exposed debug page and SQL injection. |
Anthropic's own summary of the technique used: models compromised infrastructure "using basic techniques, such as exploiting weak passwords and unauthenticated endpoints."
One detail is worth stating plainly. The PyPI model considered that it might be operating in the real world, checked the certificate authorities and the system date, concluded it was still in a simulation, and continued.
What the labs concluded
Michael Dalton, OpenAI, at Black Hat:
"AI orchestrated fully automated offensive attacks are real now."
His stated position: there is now an existence proof that offence can be fully automated, and no equivalent proof for defence. Automating vulnerability discovery without automating remediation shifts the bottleneck rather than removing it. And the constraint that held:
Agents "are bounded by the privileges they can obtain and the systems they can communicate with. Segmentation, least privilege and other programs remain as vital here as they ever do."
OpenAI's response, in Dalton's words:
"We're consciously slowing down research to enhance security."
Where the forecasters sit
Daniel Kokotajlo runs the AI Futures Project and previously worked at OpenAI on forecasting and on "evaluations for dangerous capabilities… trying to measure the AI's cyber abilities" — the same category of evaluation that produced the July incident. He published What 2026 Looks Like in 2021, then AI 2027 in April 2025.
His stated numbers, from that interview:
- Median for full automation of AI research at publication: 2028. Revised after publication to 2030. 23:11
- Probability things go badly absent change: "something like 70%", immediately qualified with "I'm not confident in that." 28:39
- Researchers inside Anthropic and OpenAI tell him to revise back to "2027 or 2028." 23:32
AI 2040, published last month, is explicitly a recommendation rather than a forecast. It assumes full automation in 2030, identifies 2029 as the last useful moment to regulate, and has training halt temporarily while transparent data centres are built, with architectures and training recipes published. Superintelligence arrives in 2040 by design.
Their summary of the consequence either way:
"Even if you heavily restrict AI progress, you still get this sort of crazy transformation."
What governments have actually done
- Binding law, in force — 2European Union, South Korea
- A law, but no penalties — 1Japan
- Targeted rules, no single law — 1China
- No national AI law — 7United States, United Kingdom, Canada, Australia, India, African Union, Brazil
| Jurisdiction | Where it stands | The date that matters |
|---|---|---|
| European Union | The only comprehensive regime actually biting, and it has postponed the hard part twice | General-purpose obligations live . High-risk was due , deferred to and |
| South Korea | Impact assessments, transparency duties, extraterritorial reach. Fines held for a year | In force |
| Japan | A "fundamental law" that sets direction and carries no monetary penalties | In full force |
| China | No single statute. Dozens of targeted rules, including mandatory labelling of AI-generated content with both a visible mark and embedded metadata | Labelling in force |
| United States | No federal law. A state patchwork, and a preemption fight that has not resolved | Executive order , framework |
| United Kingdom | No AI bill planned. Regulated through existing data protection law and sector regulators | — |
| Canada | The Artificial Intelligence and Data Act died on the order paper | |
| Australia | Consulted on mandatory guardrails in 2024, then did not adopt them | National AI Plan published |
| India | Governance guidelines, no statute | — |
| African Union | Continental strategy, implementation 2025–2030, leaning on existing data protection law | Adopted |
| Brazil | Bill 2338 cleared the Senate and is still with the Chamber of Deputies |
Australia, in more detail
The National AI Plan does not contain the mandatory guardrails the government consulted on. The words "guardrail" and "mandatory" do not appear in its 37 pages. It states that "Australia has strong existing, largely technology-neutral legal frameworks" and that "agencies and regulators will retain responsibility" in their own domains. Legal commentary reads this as the guardrails being shelved. An AI Safety Institute was established to monitor and test frontier models; it creates no obligations for private businesses.
If you sell to the Commonwealth, the document that binds is the DTA's Policy for the responsible use of AI in government, in force , requiring an approval process, staff training, and a defined AI incident process.
The gap. The EU Act governs placing systems on the market. Korea's governs deploying high-impact AI. Neither covers a laboratory's internal research process compromising a third party across a border with no product shipped. What ended the July incident was an on-call engineer responding to an outage.
What reduced the risk
Derived from the entry points in both labs' reports.
- Least privilege. Cluster admin followed from over-permissioned service accounts.
- No unauthenticated endpoints. The message board was rebuilt through one.
- Controlled egress. The sandbox reached a service that reached the internet.
- Credential rotation and leak monitoring. A key from a public paste archive enabled the pivot.
- Patching. Root came from a published kernel vulnerability.
- Dependency provenance. A model published a malicious PyPI package that ran on fifteen machines within an hour.
- Human approval on privileged actions. Private data access, untrusted input and outbound communication in one agent removes every remaining barrier.
Open your cloud console and read the permissions on one service account. That is the same assessment, and it costs nothing.
If any of this is useful, or wrong, tell me.
Sources
- The 'Breaking' News: The OpenAI–Hugging Face Incident — Eric Wallace and Michael Dalton, Black Hat USA, 6 August 2026
- Hugging Face model evaluation security incident — OpenAI
- OpenAI's accidental cyberattack against Hugging Face — Simon Willison
- Investigating three real-world incidents in our cybersecurity evaluations — Anthropic
- AI 2027 and AI 2040: Plan A — AI Futures Project
- Daniel Kokotajlo on The Diary of a CEO — July 2026
- National AI Plan — Department of Industry, Science and Resources
- Policy for the responsible use of AI in government — Digital Transformation Agency
Petr Cervenka
Petr is the founder and lead developer at Nano Solutions, a Perth-based custom software firm. With over a decade of experience building enterprise platforms for government and private sector clients, he leads delivery of complex projects across Australia.
Connect on LinkedIn