The people building the most powerful artificial intelligence systems in human history would like you to know they are a little worried about the most powerful artificial intelligence systems in human history. OpenAI chief scientist Jakub Pachocki published a statement Sunday saying that racing forward "at all costs seems absurd once one internalizes the seriousness of the stakes." The man helping build the thing is telling us the thing might be too serious to keep building the way they're building it. Totally fine.
The Alarm Is Coming From Inside the Lab
According to Axios, AI executives and researchers are increasingly vocal about a paradox that should keep everyone up at night: they believe they are in a race they cannot safely slow down on their own, and they are now asking governments, competitors, and outside institutions to force the entire field to pump the brakes.
Read that again. The people who built the accelerator are asking someone else to find the brake pedal. They've looked around the car, confirmed there isn't one on their side, and are now leaning out the window hoping a traffic cop shows up.
Pachocki's statement, reported by Axios, is striking not for being alarmist but for being honest. This is OpenAI's chief scientist, not some fringe academic. When the person responsible for the internal science of the world's most prominent AI company writes publicly that pressing ahead without restraint is "absurd," that word is doing a lot of work.
The Race Nobody Can Quit
Here is the core problem, and it is genuinely maddening. No single lab can unilaterally stop. If OpenAI slows down, Anthropic doesn't. If both slow down, Google DeepMind doesn't. If all three slow down, xAI doesn't. And if all four slow down, the Chinese government absolutely does not. The competitive logic is airtight and it points straight off a cliff.
This is what economists call a collective action problem and what the rest of us call a disaster waiting to happen. Every individual actor has strong incentives to keep running even as the group collectively agrees that running might be catastrophic. It's the nuclear arms race with worse vibes and a faster development cycle.
What makes this moment different from previous rounds of AI safety hand-wringing is who is doing the wringing. These aren't outsiders or critics or worried philosophers. These are the researchers doing the actual work, publishing the actual papers, shipping the actual products. When they start writing public statements about internalizing the seriousness of the stakes, you have to wonder what exactly they are seeing inside those labs that they are not saying out loud.
What They're Actually Asking For
Axios reports that researchers like Anthropic's Jacob Coxon are joining this call, urging governments and outside institutions to step in and impose cross-industry restraint. The ask, stripped down to its bones, is for some kind of external governance structure that doesn't currently exist in any meaningful form.
The United States government, for its part, has spent the last several years lurching between two positions: doing nothing because regulation might slow innovation, and doing nothing because nobody in a position of power understands the technology well enough to regulate it. Congress has held hearings. There have been executive orders. The net effect on the actual pace of AI development has been roughly zero.
Europe has moved further with the EU AI Act, but regulatory frameworks designed for today's systems may not be remotely adequate for whatever these labs are building right now. The gap between what regulators understand and what researchers are actually working on is widening every quarter.
Why This Matters More Than the Last Ten AI Safety Stories
The AI safety debate has been running long enough that a lot of people have tuned it out. There's a version of this story that has been written roughly every eighteen months since at least 2015, and so far the world has not ended, which makes it easy to treat each new warning as more crying wolf.
But the people issuing these warnings used to be mostly academics and think-tankers with varying degrees of credibility and access to actual systems. Now it's the chief scientist at OpenAI. That's a different category of source.
The phrase "artificial general intelligence" is hovering over all of this without being named directly in Pachocki's statement. AGI, the hypothetical point at which AI systems match or exceed human-level cognition across essentially all domains, has been the theoretical goalpost for decades. The fact that AI lab leadership is now publicly writing about the "seriousness of the stakes" in language this unguarded suggests the goalpost may not be as theoretical as it used to be.
What Happens If Nobody Listens
The uncomfortable answer is that the race continues at exactly its current pace. Labs keep hiring, keep training, keep shipping. Governments keep scheduling hearings. Think tanks keep publishing frameworks. And the researchers keep writing increasingly urgent public statements that get covered as news stories and then scroll past.
The structural incentives haven't changed. Capital is still flooding into AI at a historic rate. The competitive pressure between labs is still intense. The geopolitical dimension, with the United States and China both treating AI supremacy as a national security priority, makes unilateral restraint look not just economically foolish but strategically suicidal to anyone making decisions in Washington or Beijing.
So what OpenAI's chief scientist is really describing, whether he means to or not, is a system that has no off switch and whose builders are now publicly admitting they'd like one. The request for external governance is genuine. The question is whether any institution on earth currently has the speed, authority, and technical understanding to actually provide it. The answer, as of this September, is pretty clearly no.
The Dingo Take
You are supposed to find this reassuring. The narrative the industry keeps trying to sell is that the presence of safety researchers inside these labs is proof that responsible development is happening, that the people building these systems are thinking hard about the risks, and that we should trust the process. OpenAI's chief scientist publishing a statement describing unconstrained racing as "absurd" is not reassuring. It is the opposite of reassuring. It is a man strapped to a rocket saying he hopes someone invents a parachute soon.
The ask for government intervention deserves to be taken seriously, but let's be honest about who we're asking. The United States Senate could not get a coherent answer about how Facebook's algorithm worked in 2018. We are now asking that same institutional apparatus to regulate systems that the people who built them are describing with barely concealed alarm. Congress moves in years. These labs move in months. That gap is the entire problem, and no amount of earnest testimony is going to close it.
The darkest read on all of this is that the public statements are the point. That by going on record saying "we are worried, please regulate us," the labs get credit for responsibility without actually having to slow down. They get to be the good guys in the story while the governments they're appealing to fail, predictably, to act in time. Maybe that's cynical. But these are trillion-dollar companies making a public show of asking for oversight they know they're not going to get. The cynic in the room has been right about this industry before.
Comments