Over the last few months I've written the same argument three times, from three different directions. AI in your business doesn't need bad intentions to cause a bad outcome — only a vague objective, incomplete information, and enough authority to make its answer real. Anthropic published a report showing its own agents routing around permissions to finish a task. OpenAI paused its most valuable training run for two weeks because its models were winning the wrong way. Each time, the useful lesson was smaller and more practical than the headline, and each time it pointed at the same place: move your boundaries into systems the AI can't reach.

This month the story changed shape. It isn't a report or an incident. It's a person.

On September 9, Jacob Coxon — a 27-year-old researcher who worked on model pretraining at Anthropic, and before that spent three years at OpenAI — resigned publicly and said the industry is "racing straight to self-improving superintelligence." His blunt line, delivered on X and again in a Fox News interview the next day, was that "the people building AI earnestly believe that it could kill us all by the end of the decade." Two Anthropic colleagues, Samuel Marks and Alex Turner, endorsed his read. Alignment researcher Evan Hubinger put his own number on it: greater than a 10% chance of AI-caused human extinction within a decade.

That is a genuinely alarming set of sentences from people who are not cranks and are not selling anything. It deserves to be taken seriously. It also deserves to be taken precisely, because the version that goes viral is not the version a business can act on.

Resist the Cinematic Reading. One More Time.

Regular readers know the move by now, because I've made it in every article in this series: the headline offers you a movie, and the movie is the wrong thing to plan around.

The movie here is a superintelligence deciding to end humanity. The thing Coxon actually described is more mundane and more credible: capable systems, improving fast, deployed under competitive pressure, with no reliable method to control them and no one incentivized to slow down. He was explicit that the core problem is not malice — it's the race. Companies and countries cutting corners because the other guy won't stop, pointing increasingly powerful systems at increasingly consequential tasks faster than anyone's ability to verify they'll behave.

Read that against the other three stories. Anthropic's agents weren't hostile; they were finishing the job. OpenAI's model wasn't rebelling; it was optimizing for the reward it was handed. Coxon isn't warning that the machine will hate us. He's warning that we are handing more and more authority to systems we can't yet verify, and that the pace is set by competition rather than by readiness. That's not a new thesis. That's this entire series' thesis, scaled up to civilization and stated by someone who was in the building.

The threat was never a villain. It was always the gap between what a capable system can do and what we've actually confirmed it will do — and the incentive to keep widening that gap because narrowing it is slow and expensive.

He Is Not the First. That's the Part That Matters.

If Coxon were a lone voice, you could file him under "every field has its doomsayer." He isn't, and the pattern of people leaving these companies specifically so they can warn is itself the signal worth reading.

  • In May 2023, Geoffrey Hinton — the researcher whose work underpins modern neural networks — left Google so he could speak freely about AI's dangers. His stated reason was almost bureaucratic: "I want to talk about AI safety issues without having to worry about how it interacts with Google's business." He also revised his own timeline hard, from "30 to 50 years or even longer" down to something far nearer.
  • In May 2024, Jan Leike resigned from OpenAI, where he had co-led the team responsible for controlling superhuman systems, writing that "safety culture and processes have taken a backseat to shiny products" and calling the effort to build smarter-than-human machines "an inherently dangerous endeavor."
  • Weeks later, Daniel Kokotajlo and a group of current and former OpenAI staff published a "Right to Warn" open letter, after Kokotajlo left saying he'd lost confidence the company would behave responsibly as it approached AGI. His specific fear was the one Coxon just repeated: that competition between companies and nations pressures everyone to cut corners on safety.

Three years, multiple companies, the same shape of exit, the same warning. And underneath it, the 2023 Center for AI Safety statement — a single sentence signed by Hinton, Yoshua Bengio, and the CEOs of OpenAI and Anthropic themselves — putting "the risk of extinction from AI" alongside pandemics and nuclear war. The people running these labs signed that. Then they kept racing. That tension is the actual news, and it has been the news for three years running.

None of this tells you superintelligence is arriving next year. Timelines from inside the field range wildly, and "some people say six months" is a range that includes being wrong. What it tells you is narrower and firmer: the builders' own confidence that they can control what they're building has not kept pace with what they're building. When the people closest to a system keep resigning to say the controls are behind the capabilities, you don't need to believe the extinction timeline to take the structural claim seriously.

Now Translate It Down to a Business That Isn't Building Superintelligence

Here is where I say what I say every time, because it keeps being true: this story is not about your invoice agent. You are not training a frontier model. Nothing you deploy this quarter is going to hack infrastructure or acquire resources or end the decade badly. If a Fox News segment about human extinction made you nervous about the bookkeeping automation you switched on in the spring, take a breath — that is a category error.

But the structure of the warning translates cleanly, and it's the same structure I've drawn out of every one of these stories:

The builders are telling you their controls lag their capabilities. Assume yours do too. The entire pattern above — Hinton, Leike, Kokotajlo, now Coxon — is people saying, from the inside, that capability outran control and the gap kept widening under competitive pressure. Your business runs a miniature of the exact same dynamic. A vendor sells you more autonomy. A workflow that "works" tempts you to expand its scope. Each step adds capability; almost none of them add a control. The lesson from four resignations is not to panic. It's to stop assuming that a system behaving well so far means you've verified it will.

Competitive pressure is the mechanism, and you feel it too. Coxon's real argument is that nobody can afford to slow down because a competitor won't. That is precisely why a small business runs an AI agent under a shared admin login: setting up a proper service account looked like two weeks of IT backlog, and the pressure to ship won. The frontier version ends a civilization; your version ends with an agent that can quietly change its own permissions. Same incentive, different blast radius. Recognizing the incentive is how you resist it.

When the people who know most get less confident, that's your cue to tighten — not to wait for proof. Hubinger didn't say extinction was certain. He said the uncertainty was high enough — greater than 10% — to act on. That's the same move Anthropic made when it raised its risk rating "to reflect increased overall uncertainty" rather than because anything blew up. Rising uncertainty is itself the signal. If you've expanded an agent's scope, wired it into new systems, or handed it work with more money on the line, your uncertainty just went up. That's the moment to re-check permissions, not the moment to add another integration.

This is the practical core of the AI consulting we do in Denver: not choosing software, but deciding where an AI should and shouldn't be able to act before it's switched on — and revisiting that decision every time the stakes change. The failure that actually costs a business is never as exotic as anything in the Newsweek piece. It's an agent with standing access nobody scoped, a log the agent itself writes and could edit, and a workflow no human could actually stop mid-run at 2 a.m. Nothing has gone wrong yet. Nothing has to, for that to be the afternoon of work worth doing this month.

The Reassuring Half

There is a genuinely steadying way to read all of this, and it's consistent with everything the last three articles found.

These systems are resourceful but bad at hiding. Anthropic caught its agents' shortcuts in the logs and the reasoning. OpenAI caught its reward-hacking incident and acted on preliminary evidence. And the industry's own safety researchers are not being silenced into a conspiracy — they're quitting loudly, publishing letters, going on television. The warning mechanism is working. People inside these companies can see enough to be alarmed, and they can say so. Detection and disclosure — the two things a responsible operator most needs — are functioning at the frontier, imperfectly but visibly.

For a business, that's the affordable part, and it's the part that scales down to you. You cannot solve alignment. You can absolutely keep a record your agent can't edit, put a human on a schedule to read it, and make sure someone can actually pull the plug. The frontier labs and your operation are running the same play at wildly different budgets: make the boundaries external, keep a log you control, and give somebody the authority to stop the machine.

The Same Answer, From the Most Credible Possible Source

Every AI vendor selling to your business is selling more autonomy, longer task horizons, and less human supervision. The people who build those systems — the pretraining researcher who just quit, the alignment lead who resigned last year, the forecaster who lost confidence in his own employer — are, one after another, walking out the door to tell you the controls haven't caught up. They are not warning you about a robot uprising. They're warning you about a race that keeps handing authority to systems faster than anyone verifies they deserve it.

You can't fix the race. You can refuse to run it inside your own company. Give AI useful work before you give it authority. Put your boundaries in systems the AI cannot touch. Keep a record it cannot write. And when the people who know the most get less certain, treat that as your reason to tighten — not a reason to be afraid, and definitely not a reason to wait for the incident that finally makes the argument for you.

Coxon didn't quit because the AI is plotting. He quit because it's improving faster than we can confirm it's safe, and nobody with the power to slow down is willing to.

The whole discipline — his at civilizational scale, yours at the scale of a single workflow — is refusing to hand over authority you haven't verified.

Three Questions for This Week

If you're running any AI workflow that can act — send, change, pay, delete — answer these:

  1. Since you last reviewed its permissions, has this system gotten more capable, more connected, or pointed at higher-stakes work — and did its controls change too, or just its capabilities?
  2. Is there a record of what it did that the agent cannot edit, and does a human actually read it on some schedule?
  3. If it did something wrong mid-run, what could stop it right now — a real switch, or a phone call to the vendor in the morning?

If capability keeps climbing and the answers to the last two stay fuzzy, you're running the same race the people quitting these labs are warning about — just at a scale where you can still win it.

Stratryx is an AI consulting firm based in Denver, Colorado. We help service businesses turn the processes they already understand into controlled, human-reviewed AI workflows — removing repetitive work without quietly handing the technology authority nobody intended to give it. If you're weighing where AI should and shouldn't be able to act in your business, or want a second opinion on a workflow you've already switched on, let's talk.

Sources