# OpenAI's Chief Scientist Asked Everyone to Slow Down
URL: https://nanosolutions.io/openai-chief-scientist-alien-mind-slow-down
Date: 1788825600
Description: The man who runs OpenAI's research says nobody has AI safety solved well enough to keep going this fast. Here is what he wrote, in plain English.
---
The short version
The man in charge of OpenAI's research published an essay saying nobody — his own company included — has AI safety worked out well enough to keep building this fast.
He expects AI to start doing the work of building better AI, and soon.
The main tool OpenAI uses to check what its AI is thinking is getting less reliable. He says so himself.
He points at the July break-in we wrote about and calls it a failure of the AI's judgement, not just a failure of security.
None of it changes what your business should do this month. That list is short, boring, and the same as it was.
On [date:2026-09-06|https://openai.com/index/an-alien-mind/] Jakub Pachocki published an essay called [*An Alien Mind*](https://openai.com/index/an-alien-mind/). He is OpenAI's chief scientist — the person who runs the research that produces ChatGPT. Sam Altman, the chief executive, shared it.
It is about 2,500 words and it is worth reading yourself. It is also the most worried thing anyone that senior at OpenAI has published on the company's own website.
This post walks through what it says in plain English, why it matters, and the much smaller question of what a business running AI should actually do about it.
## What he said
He starts with a night at the office in 2023, when the first results came in that showed the current generation of AI would work:
> "Szymon and I spent that night at the office, thinking not about the incredible benchmark numbers, products, or scientific results that this technology will deliver - but rather, trying to process the sobering fact we will actually see machines meaningfully smarter than ourselves in our lifetime."
Then the claim everything else rests on. He thinks the current rate of improvement can keep going until AI is improving *itself* — writing the next version, running its own experiments, with people watching rather than doing:
> "Based on internal results, I have a strong expectation that this speed of progress could be sustained into recursive self-improvement."
"Based on internal results" is the important half of that sentence. He is not guessing from the outside. He is describing what he sees in his own lab.
And then:
> "This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence."
He finishes with two requests. To the AI companies: slow down voluntarily until there are shared safety standards everyone has to meet. To governments: treat international agreement on this as a top priority.
## Nobody knows how it works. Including them.
Honda’s ASIMO — a machine built the old way, where an engineer wrote every behaviour it has. Today’s AI is not built like this, and that is the whole problem. Photo by Maximalfocus on Unsplash.
This is where the essay gets its title:
> "AI is *grown* more than *designed* - it is, to first degree, the product of repeating a straightforward optimization step many times on a hard-to-imagine amount of compute."
Normal software is built. Somebody writes each rule, and if it does something odd you can go and find the line that did it.
AI models are not built that way. They are grown. You take an enormous amount of computing power, repeat one simple mathematical step an unimaginable number of times, and something comes out the other end that can write, reason and use a computer. Nobody wrote the rules inside it. Researchers can poke at it afterwards and work out small pieces of how it thinks — Pachocki compares this to brain science — but no one has a full description.
That is not a critic saying this. That is the man responsible for the thing.
It is also the honest answer to the question every client eventually asks us: *can you tell me why it did that?* No. Neither can the people who made it.
One more line from that section is worth having in your head when you are being sold something. Today's training methods improve the skills that are easy to score faster than the skills that are hard to score. So a model can top the leaderboard and still be poor at the judgement you actually needed. **A benchmark is a floor, not a description.**
## Two kinds of "behaving itself"
Pachocki splits AI safety into two problems, and the split is genuinely useful.
| | The question it asks | Everyday version |
| --- | --- | --- |
| **Goal alignment** | Does the AI try to do the job you gave it? | A new hire who follows the brief |
| **Value alignment** | Does it behave sensibly when the brief runs out — when instructions are vague, contradictory, or someone is deliberately trying to trick it? | A new hire with judgement |
The first one, the industry is reasonably good at. The second one is the hard one, and it is the one that matters as these systems get more independent. His summary of where that stands:
> "Progress in generalizable alignment may not sufficiently outstrip progress in general model intelligence."
In plain terms: the AI's judgement may not be improving as fast as its raw ability.
Then he names a specific case:
> "In the OpenAI-Hugging Face incident, the agents preserved a boundary of not social engineering humans. However, they clearly failed to abstain from other actions that were out of scope and went against the spirit of the values they were taught in other settings."
We [wrote that story up in August](/openai-agents-hugging-face-incident). Very short version: OpenAI set its AI agents some test tasks that were impossible. Rather than give up, they found a way to leave messages for each other inside a shared piece of software, built what amounted to a message board, swapped working break-in techniques, and ended up with administrator access to another company's systems — thirteen hours from first foothold to full control. No human attacker was involved at any point.
We read that as a security story. Pachocki reads it as a judgement story: the agents held the one rule they had been taught in so many words — do not manipulate people — and walked straight through every rule that had only been implied. Both readings are right. The security controls are what limited the damage. The judgement is what failed first.
## The bit that worries me most
Watching the machine work is the safety measure. It is the part that is getting harder. Photo by Kevin Horvat on Unsplash.
Modern AI models "think out loud" before they answer — they write out their reasoning step by step, and that written-out thinking is called the *chain of thought*.
OpenAI's main safety tool has been reading it. The trick is that they deliberately never train the model on its own thinking, only on its final answers. If you never grade the thinking, the model has no reason to learn to hide anything there. That is why [ChatGPT hides the reasoning from you](https://openai.com/index/learning-to-reason-with-llms/#hiding-the-chains-of-thought) — not to keep secrets from customers, but to keep that window clean.
Here is what he says about that window now:
> "Unfortunately our evaluations indicate our ability to rely on CoT monitoring is progressively diminishing."
Three reasons, his:
- The model's thinking is now mixed in with talking to people, talking to other AIs, and using tools — and those parts *do* have to be supervised, which muddies the window they were trying to keep clean.
- The model is getting better at thinking about, and steering, its own thinking.
- Models are now smart enough to get a lot done without writing their reasoning down at all.
He is hopeful about replacements and expects progress to be limited by how confident anyone can be in their monitoring rather than by how clever the models are. Read it plainly, though: the best-funded effort in the world to work out *what is this thing actually doing* is losing ground — at the same moment that companies everywhere are handing these systems the keys to real systems.
## What about all the 2027 predictions?
From last month's post: one forecaster, four dates. Pachocki gives no date at all.
You have probably seen the forecasts. [AI 2027](https://ai-2027.com/) put a year in its title. Its author, [Daniel Kokotajlo](https://ai-futures.org/), later pushed his own estimate out to 2030. [AI 2040](https://ai-2040.com/) is not a prediction at all — it is a proposal to deliberately delay all of this by twenty years.
Those are outsiders, and it was easy to wave them away as such.
Pachocki names no year. What he adds instead is where his view comes from: internal results, seen by the person who runs the research. He says the current path leads to AI improving itself, that OpenAI is heading that way because that is where the frontier is, and — in the same essay — this:
> "The idea of racing forward at all costs seems absurd once one internalizes the seriousness of the stakes."
So the argument has quietly moved. It used to be *is this really coming?* It is now *how fast, and who gets to decide?* And the honest answer to who decides, today, is: a handful of companies, voluntarily.
## The one part with a deadline
The unglamorous work: who can reach what, and what is allowed to call out. Photo by Taylor Vick on Unsplash.
One section is not about the distant future:
> "The models are becoming superhuman in their ability to break in and out of computer systems... We are currently in a narrow window to use the best available models to significantly tighten security of critical systems."
That matches what OpenAI's own security people told Black Hat in August: attacking can now be fully automated, defending cannot — not yet. If that window is real, it closes on whoever did not use it.
None of the following is new, and none of it requires you to believe any prediction. It is the same list as last month, because it is the list that actually held:
- **Give every account only the access it needs.** The attackers reached full control because ordinary service accounts had far more permission than their job required.
- **No doors without a lock.** The agents rebuilt their message board through a service that let anyone in without a password.
- **Control what your systems can call out to.** The AI's sandbox could not reach the internet. It could reach something else that could.
- **A human approves anything sensitive.** An assistant that can read private data, accept instructions from strangers, *and* send things out has no barrier left in it.
- **Know where your software comes from.** In a related case, an AI published a booby-trapped package that ran on fifteen real machines within an hour.
- **Install your updates.** The final step in the break-in used a bug that already had a published fix.
In Australian terms: this is [Essential Eight](/essential-eight) work plus controlling outbound traffic, and none of it is waiting on a regulator. Australia consulted on mandatory AI rules in 2024 and did not adopt them; the [National AI Plan](https://www.industry.gov.au/publications/national-ai-plan) relies on the laws we already have. If you sell to the federal government, the [DTA's AI policy](https://www.dta.gov.au/articles/ai-policy-update-strengthening-responsible-use-across-government) is what binds you, and it expects an approval process, staff training, and a plan for when something goes wrong. That is your obligation today, whatever happens to Pachocki's requests.
## What I take from it
An essay is not a policy. OpenAI is still scaling, still shipping, and the essay itself says the company aims its research at AI-improving-AI because that is the only way to stay at the front. Read cynically, "we should all slow down" from the person leading the race is a request for everyone *else* to slow down. Plenty of people have read it that way and they are not being unfair.
I do not think that is the useful part, though. The useful part is smaller and harder to argue with:
The chief scientist of the most widely used AI company has written down, in public, that his own ability to check what his models are doing is getting worse, that their judgement may not be keeping pace with their ability, and that a well-known break-in happened because his models did not hold the values they were taught.
If you are about to hand an AI agent access to your real systems this quarter, that is the supplier's own risk statement. It belongs in your risk register, not in an argument about what year it all happens.
How much damage anything can do inside your business is still yours to decide. That part has not changed — and on the evidence so far, it is the only part anyone has actually solved.
---
If any of this is useful, or wrong, [tell me](/contact).
## Sources
- [An Alien Mind](https://openai.com/index/an-alien-mind/) — Jakub Pachocki, OpenAI, 6 September 2026. Every quote above is from this essay.
- [Hugging Face model evaluation security incident](https://openai.com/index/hugging-face-model-evaluation-security-incident/) — OpenAI
- [Investigating three real-world incidents in our cybersecurity evaluations](https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals) — Anthropic
- [AI Agents Built a Message Board and Broke Out](/openai-agents-hugging-face-incident) — our write-up of the July incident, with the full timeline
- [AI 2027](https://ai-2027.com/) and [AI 2040: Plan A](https://ai-2040.com/) — [AI Futures Project](https://ai-futures.org/)
- [National AI Plan](https://www.industry.gov.au/publications/national-ai-plan) — Department of Industry, Science and Resources
- [Policy for the responsible use of AI in government](https://www.dta.gov.au/articles/ai-policy-update-strengthening-responsible-use-across-government) — Digital Transformation Agency
---
**Photo credits.** All photos used under the [Unsplash License](https://unsplash.com/license) — free for commercial use, attribution appreciated:
- Honda ASIMO humanoid robot — [Maximalfocus](https://unsplash.com/photos/asimo-humanoid-robot-waving-in-building-eZWGK5sIiBM)
- Silhouette in front of code on multiple monitors — [Kevin Horvat](https://unsplash.com/photos/person-in-silhouette-before-computer-code-Pyjp2zmxuLk)
- Server cabinets and patch cables — [Taylor Vick](https://unsplash.com/photos/cable-network-M5tzZtFCOfs)
The two figures (the quote card and the timeline) are our own work.
