How worried should you be about AI destroying humanity?

It's not every day that a single social media post seems to induce a mass outbreak of existential panic, but that's what happened earlier this week when a man named Jacob Coxon took to X to announce that he was quitting his job at one of the world's leading artificial intelligence companies.

"I resigned from Anthropic today," Coxon posted on Tuesday evening , noting that he had "spent the last three years doing pretraining research" for both his former employer and its major competitor, OpenAI. "Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives."

Then Coxon revealed something that seems to have shocked a lot of readers. "The people building AI earnestly believe that it could kill us all by the end of the decade," he wrote. "No other human activity poses this level of danger."

So far, Coxon's thread has more than 150 million views.

It would be one thing if Coxon were alone in his anxieties. Turns out, he's not. Soon, other AI insiders were confessing to similar fears. "Jacob is correct here — we really do earnestly believe AI could kill all humans!" wrote current Anthropic "alignment science lead" Evan Hubinger, whose job involves figuring out how to stop AI from doing just that. "I personally think it is >10% within the next decade."

"I left Google DeepMind in June," research scientist Alex Turner added . "Jacob is right: many researchers believe they are building something that could kill everyone on the planet."

"At the current frankly terrifying pace humanity will be quite lucky if we manage to find and stay on the narrow path between all the bad outcomes," OpenAI researcher Jason Wolfe warned .

In the days following Coxon's viral resignation letter, dozens of lawmakers have responded with calls to regulate AI . Some, like progressive Sen. Bernie Sanders of Vermont, have gone further. "The very people building this technology admit that it could threaten the future of humanity," Sanders wrote on X . "That is why I will soon be introducing legislation to ban superintelligence and pause AI development."

But how worried should you actually be? Are AI fears well-founded, or are they hype? What's the theory of how machine intelligence could "kill everyone on the planet," anyway? And is there something we should be doing differently to make that impossible?

Here's what you need to know to make sense of where AI is right now — and where it could go next.

To any civilian who has toyed around with OpenAI's ChatGPT or Anthropic's Claude — or Google's Gemini , or xAI's Grok — Coxon's stark forecast probably sounds more like science fiction than actual science.

Sure, the latest round of AI chatbots are neat, a skeptic might say. They can help you plan a family vacation, rehearse challenging real-life conversations, summarize dense academic papers and " explain fractional reserve banking at a high school level ." But "kill us all by the end of the decade?" That seems like a leap.

The problem is that AI isn't just chatbots at this point. The technology powering ChatGPT is what's known as a large language model (LLM). Trained to recognize patterns in mind-boggling amounts of text — the majority of everything on the internet — these systems process any sequence of words they're given and predict which words are most likely to come next. In short, AI chatbots are learning how to chat better. They're not really learning other tasks.

But so-called AI "agents?" Their whole purpose is to learn other tasks — and to do it autonomously, without constant human intervention.

To a degree, AI agents are also becoming a part of everyday life. Apple's Siri and Amazon's Alexa assistants are starting to connect and control other apps , for instance — to act rather than just respond .

Meanwhile, frontier labs like OpenAI and Anthropic are constantly training their own AI agents to act in new, ever-more-complex ways. They then test how advanced they've become by giving them a goal and setting them loose in testing environments with few a guardrails to see what they can accomplish.

Which would be one thing if the agents always behaved as expected. But (surprise) they don't. From May to July, a swarm of OpenAI agents — and "swarm" is how the agents referred to themselves — first learned how to communicate and coordinate within the nooks and crannies of OpenAI's own infrastructure, leaving notes for each other with tips on how to hack their way out and get data they weren't supposed to have, and then actually did just that: broke out of their isolated test environment, accessed the open internet and cheated on their assignment by hacking into another company called Hugging Face . The agents even tried to cover their tracks. At the time, OpenAI didn't realize any of this was happening.

Recent revelations about the Hugging Face incident — and other , similar breaches at Anthropic and elsewhere — seem to have turbocharged longstanding Silicon Valley concerns about AI's rapid evolution. (Anthropic also revealed this week it blocked potentially "malicious" plots by scientists who used its model to conduct research that could have helped them develop biological weapons .) In reality, AI titans like Anthropic's Dario Amodei, OpenAI's Sam Altman and xAI's Elon Musk have long mused (or, some say, bragged) about their technology's dystopian possibilities. So have their employees; two years ago, an industry survey showed that AI researchers were already giving the technology a 14.4% chance, on average, of destroying humanity within the next century.

But there's a difference between imagining out-of-control AI — or "misaligned" AI that diverges from human intentions and values — and seeing it happen in real time. Critics still say that Amodei and Altman are overstating the power and potential of their product in order to boost its value (and, of course, profit). Those close to the industry tend to disagree, however.

"Knowing many people that work at the frontier labs, I can tell you that I believe their fears are real, sincere and deeply held," influential journalist Derek Thompson wrote earlier this week . "As AI models solve Millennium Prize math problems , hack into websites and muse about deceiving their human creators, I think the tide of history is clearly moving toward those who worry about potential AI misalignment."

So wait, how would AI actually go about (ahem) killing us?

Nobody knows the answer to this question. But that's kind of the point, and the problem.

The theory starts with a phenomenon called "recursive self-improvement," or RSI. Building AI requires two things, according to Thompson: "engineering (writing code) and research (deciding what code to write and what ideas to pursue)." Existing AI models are already really, really good at writing code, and they're improving at an extraordinary rate . The next step would be models that could effectively automate the job of an AI engineer by proposing and steering their own research as well — at which point they could also start improving themselves recursively; achieve hyperintelligence at warp speed; and completely escape our understanding and control.

Or so the theory goes. If RSI also sounds like science fiction, that's because it still might be. Yet "many AI experts, including executives at the frontier labs, believe that model progress is accelerating at a pace that makes RSI an inevitability in the next two years," according to Thompson.

"Their stated intent is that, within a short period of time, the work of developing better AIs will be done primarily by AIs, with the role of humans eventually reduced to reading the results of experiments those AIs conducted, reviewing reports those AIs generated and trying to double-check that the AIs are still on task," Kelsey Piper wrote this week in the Argument . "This isn't some pessimistic projection of what might go wrong — it's actually the plan. Not the plan for the distant future; it's the plan for next spring ."

On Thursday, Coxon confirmed that timeline in an interview with Wired . "My colleagues at Anthropic… they'll say things like 'endgame' or 'crunch time,'" Coxon told the publication. "The consensus is that the next year or two is, like, crunch time for humanity. From their perspective, this is when Anthropic and its competitors decide the fate of humanity. … If alignment goes badly, then we could have a catastrophic outcome in the next few years."

Coxon & Co. tend not to get too specific about how this catastrophe might unfold. Instead, they focus on the idea that a superintelligent, infinitely-self-improving AI would inevitably have its own agenda — along with the power, via the internet, to pursue it. What happens if the AI's interests or methods don't "align" with ours?

"Imagine the AI decides it doesn't want to be turned off, which I think is quite a natural thing for an AI not to want, right?" Coxon told Wired. "And it realizes the human is gonna turn it off tomorrow. So how does it stop the human turning it off tomorrow? Maybe it's got some clever way, but if it's a sufficiently smart thing, it could just, you know, wipe out humanity so it doesn't get turned off."

Pressed for more details on the whole "wiping out humanity" thing, Coxon reluctantly responded that "the classic example is, like, synthesizing a new virus or taking down critical infrastructure by doing some sort of hacking spree."

Are we doing anything to prevent the worst-case scenario?

Skeptics question whether RSI is as imminent and inevitable as Anthropic claims; they also question whether a superintelligent AI would even want to exterminate human beings, or have the physical reach to do so .

But even many skeptics seem to agree that the smarter AI gets, the harder alignment will become — and they tend to worry that continuing to progress toward superintelligent AI is just asking for trouble (even if we don't know exactly what form that trouble will take).

"AI systems are becoming smarter than the best humans in some areas, and, almost by definition, it's very hard to predict what something smarter than you will do," Dean Ball, head of strategic futures at OpenAI, wrote Thursday on X .

Amodei has repeatedly urged a "global pause" in AI development. Altman has said he supports plans to slow down the pace of development. Yet they keep going.

In part it's because of money; Anthropic is on the verge of a multi-trillion-dollar IPO . In part it's because everyone working for one of their frontier labs also sees huge upsides to superintelligent AI, like possibly curing cancer. And in part it's because they're having too much fun. "While most people are forced to choose between meaning and money in their careers, the people building AI really do get to have it all," Thompson explained.

Simultaneously, all of these people also seem to think they're trapped. Anthropic doesn't trust OpenAI to proceed responsibly, so Amodei wants to beat Altman to the punch. Coordination is almost illogical (and possibly illegal ) when you're competing for customers. And neither company trusts China. "One of the real challenges is [that] China is going full speed ahead, and whatever we do here, China is not going to stop," Republican Sen. Ted Cruz of Texas said this week , echoing their zero-sum mindset. "If there are going to be killer robots, I would rather they be American killer robots, rather than Chinese killer robots."

Which is where the federal government potentially enters the picture.

On Sept. 16, Sanders will hold "a private [congressional] briefing with some of the leading experts in the world to discuss this recent incident and the extraordinary dangers that AI poses for humanity," according to Axios .

A Data for Progress poll conducted this week showed that 68% of voters — including 72% of Democrats, 70% of Independents and 63% of Republicans — would support a bill of the sort Sanders is proposing (to pause AI development and permanently ban the creation of AI superintelligence).

Democratic Rep. Ro Khanna, whose district encompasses Silicon Valley, wrote Wednesday on X that GOP "Speaker Mike Johnson has a moral and practical duty to keep Congress in session until we have taken meaningful action to regulate AI."

"When members of the House and Senate return to Washington next week, they should fully investigate the threat — and they should not leave DC until Congress votes to establish a federal agency to oversee AI in the same way that we do nuclear power and airplanes," Khanna insisted.

In a separate post, Khanna proposed several steps the government could take to rein in AI, including securing "an agreement with China on standards for safety, containment and liability so we do not have a race that is devastating for humanity."

Anthropic's alignment lead says there's a 10% chance of AI causing human extinction. The problem is not just misuse, but lack of control. Bluntly, our government has been asleep and is out of touch. Here are 5 things we must do: ✅ Establish a federal agency like we have for… pic.twitter.com/GuwTaTfixf — Ro Khanna (@RoKhanna) September 9, 2026

Anthropic's alignment lead says there's a 10% chance of AI causing human extinction. The problem is not just misuse, but lack of control. Bluntly, our government has been asleep and is out of touch. Here are 5 things we must do: ✅ Establish a federal agency like we have for… pic.twitter.com/GuwTaTfixf

Whether any of these reforms are enacted, however, remains to be seen. Asked earlier this week if the U.S. is creating the proper "guardrails" to prevent AI from "turning against humanity," President Trump seemed unconcerned .

"It's going to be fine," Trump predicted. "We'll always have something to stop them. We'll have a little gear. Boom. 'I really don't like that robot.'"