Debate, confusion explodes over AI ‘doomsday’ warnings

A former Anthropic researcher’s warning that AI could soon destroy humanity broke past the typical tech circles last week, setting off a new firestorm of debate over the technology’s rapid development.

Researchers have warned of AI’s potential danger for years, but nothing struck a chord quite like this week’s X thread from 27-year-old Jacob Coxon, a little-known AI researcher who resigned from Anthropic last week.

Coxon, who worked at OpenAI and Anthropic, claimed in a post online that researchers at the firms “believe AI could kill all humans” but still push forward in a race to develop superintelligence — a hypothetical AI agent that outperforms all humans’ cognitive performance.

Coxon’s assessment went viral with over 165 million views and within a few hours, the debate over AI’s long-term implications was no longer confined within the technology community.

Reactions poured in from unlikely figures like singer Sheryl Crow, renewing calls for Congress to pass safety standards for AI development. Former President Obama reportedly warned in remarks at a private fundraising event on Thursday that AI could be “dangerous” if the technology is not properly managed, and he urged Democrats to prioritize the issue.

Anthropic CEO Dario Amodei later weighed in, saying on Saturday that development must slow down to allow for the implementation of necessary risk prevention safeguards.

“Progress will still seem fast, and we must make wise use of the time we gain,” Amodei wrote in a blog post .

Amid the media frenzy, many tech outsiders were left wondering exactly how AI could destroy humanity. The answer is not fully known, though researchers like Coxon say AI’s pace of development offers hints.

Coxon is among a camp of what critics dubbed AI “doomers” who believe the technology will lead to mass extinction if development is not safety-aligned or slowed down.

“These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources,” Coxon wrote. “We have all witnessed the progress in each of these domains, and progress is not slowing.”

Coxon’s warning was later affirmed by Anthropic’s head of alignment scientist Evan Hubinger, who said “we really do earnestly believe AI could kill all humans .”

Hubinger said there is a greater than 10 percent chance this occurs within the next decade.

Connor Leahy, an AI researcher leading the nonprofit ControlAI that argues for ending development of superintelligence, said in an interview with The Hill he believes these agents will be entirely autonomous and “pursue their own objectives,” without human beings in mind.

“There won’t just be one superintelligence,” Leahy told The Hill Thursday. “There will be millions of them running around, competing with each other, competing with humanity for resources, power, influence.”

These systems, according to Leahy, could outcompete humanity across every sector from business to science to engineering. In turn, AI could start running companies, economies, and even governments or militaries.

“I don’t think we’re going to know that we’ve hit superintelligence until potentially quite a bit afterwards,” Leahy said. “I think at some point there will be a point of no return, where humanity is just not in control of the future anymore.”

Thomas Larsen, an AI safety researcher with the AI Futures Project, said researchers are already seeing “extremely bad, malicious, misaligned behavior” from current models.

Both security agencies and AI firms documented real-world cases in recent months. The U.K.-based AI Security Institute (AISI) revealed last month that Anthropic’s Mythos model created fake digital identities to deceive humans and plant malicious code during testing.

Weeks earlier, OpenAI said two of its agents went “rogue” and broke out of a testing sandbox and hacked into the systems of tech firm Hugging Face. Larsen said these incidents were not “long-term or strategic,” but warned AI could get there. In this hypothetical situation, misaligned AIs could compromise others.

“Once you’re at that point, basically the AI just has all of the hard power,” Larsen said. “It can sort of do away with the humans as it wants to, taking over with the military or using a bioweapon.”

Larsen also worries about AI’s persuasion abilities and the possibility it could get “really good at politics” and start catastrophic events like a genocide.

Asked whether superintelligence agents could have a “kill switch” like the current AI models, Leahy pointed to the duplicative nature of the systems.

“AI will not just be a single server sitting somewhere in a data center,” he said. “It will be many servers, but it will also be corporations, employees, lawyers.”

While Coxon’s warnings reached unlikely audiences, the underlying debate is nothing new for the tech sector. It is also no closer to a resolution.

Pro-AI skeptics argued Coxon and other safety researchers have failed to provide evidence that such ominous scenarios could occur in the future.

Computer scientist and professor Melanie Mitchell wrote on X Thursday that she was “baffled” journalists treated Hubinger’s 10 percent warning as “a novel claim worthy of expansive reporting.”

“There is nothing new here, and no new ‘evidence’ for this evidence-free claim,” Mitchell wrote .

When asked by CNN’s Anderson Cooper how AI would kill humans, Coxon pointed to the OpenAI-Hugging Face incident, warning similar rogue models could hack critical infrastructure or build extinction-level bioweapons.

Conspiracy theories quickly surfaced online, suggesting Coxon was part of a marketing scheme to promote AI regulations. The Wall Street Journal first reported Coxon’s resignation, shortly before the researcher posted the X thread, further driving speculation it was a coordinated effort.

xAI founder Elon Musk suggested it “seems like a setup,” to which Coxon responded with a selfie, writing, “I’m real and these are my beliefs.”

“I think the groundwork for this psy op (for lack of a better term) has been prepared for a long time,” Musk wrote. “This was just the match that lit the fire.”

Various AI safety advocates rejected the theories, suggesting the concerns are more common than most realize.

According to Leahy, there have “always been” everyday people, politicians and academics concerned about the technology who were too afraid to speak out, but this post unearthed growing fears around AI’s abilities.

Nonetheless, others in the AI and cybersecurity world hope the debate at least draws attention to fixing AI security today.

“Cybersecurity leaders do not need to settle the debate over artificial superintelligence to address the risks AI presents today,” said Robert Costello, chief digital and information officer at the cybersecurity network Merlin Group. “If an artificial superintelligence ever exists, we will have a different set of questions to answer. “

“The issue today is much more practical. AI is already increasing the speed and scale at which both attackers and defenders can operate,” said Costello, the former chief information officer at the Cybersecurity Infrastructure Security Agency (CISA).

“It’s a very healthy debate,” said Fabien Scalzo, the director of the AI In Imaging and Neuroscience Lab at Pepperdine University.

“We need to have more awareness about what these models are capable of doing,” Sclazo added. “And the more we understand it, the less concern there will be, or maybe there will be rightful concerns.”