AI staff 'genuinely frightened' for humanity's future, ex-Anthropic researcher tells BBC

An AI researcher who quit Anthropic told the BBC that people working on the technology are "genuinely frightened" about the speed of its advancements and what it could mean for humanity.

Jacob Coxon spoke with BBC's Laura Kuenssberg on Saturday about his viral resignation post where he raised concerns about out-of-control artificial intelligence.

"I believe that if we don't slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future," he said.

His resignation comes amid growing safety concerns in the industry, including from his former boss, head of Anthropic Dario Amodei, who in an essay on Saturday argued for development to slow down.

Speaking to Sunday with Laura Kuenssberg, 27-year-old Jacob Coxon - who worked at OpenAI before joining Anthropic - welcomed the suggestion of a slowdown from Amodei but said it would need to be co-ordinated with China.

"The people who work at these companies are completely serious when they ask for regulation because they find themselves trapped in a race. And they're scared of the outcomes of that race."

"Sam Altman and Elon Musk have agreed that this is a great idea. But I think there'll need to be some sort of coordinated slowdown with China if we're going to avoid a race at an international scale," he said.

The hardest question to answer, according to Coxon, is what an AI apocalypse would look like.

One of the risks outlined in Amodei's comments was of a swarm of bots acting like a supercomputer that could take over the internet.

Coxon said this scenario could be realistic in six months to a year.

In response to Coxon's departure, an Anthropic spokesperson told BBC News: "We have always been transparent that AI will bring both enormous benefits and unprecedented risks. To address these risks, we continue to build models with some of the strongest safeguards in the industry."

The firm has been a pioneer in studying how AI models work, the spokesperson added, it was the first to publish a framework for mitigating the risks posed by their development, and it also "aggressively" tests its models and publishes the findings to aid scrutiny and prevent incidents of "AI misalignment".

"This work is also why we believe the world would benefit from the industry adopting a lawful, verifiable way to work together to pace how we release powerful models," they said.

In Amodei's essay, called We Must Pace the Frontier and made public on Saturday, he proposed a three-point plan that includes independent monitoring of AI models as they are developed, industry-wide regulation and global regulation.

Dramatic insider warnings over AI fall flat with some in Silicon Valley

Anthropic and OpenAI are reportedly preparing for potentially record-setting initial public offerings. Industry figures have suggested comments about the perils and power of AI may be designed to generate hype. Other critics say Anthropic has been trying to trigger a regulatory push to block competition, leaving it and OpenAI with a duopoly.

Coxon's views have also been questioned by the CEO of the AI platform Hugging Face, Clement Delangue.

"Sorry, but asking Jacob about AI extinction risk is like asking your AC guy about climate change," he wrote on X. "Not saying it's necessarily uninteresting or wrong per se but let's keep things in perspective."

After Amodei's essay on Saturday, however, Delangue offered to help be a part of the potential solutions proposed by the Anthropic boss.

Nvidia boss Jensen Huang also discussed Coxon's comments before a crowd at a conference hosted by the investment bank Goldman Sachs last week, multiple people in the group told the BBC. They said he dismissed them as untrue.

Huang has previously said the notion that AI "is going to be the end of humanity" is "complete nonsense".

And while he may have a business interest in an AI boom - Nvidia builds chips that power AI systems - his comments reflect a growing backlash in Silicon Valley to the existential warnings from current and former staffers.

Coxon said his peers fear the danger could even arrive in the next two years.

He said staff at AI firms were "planning what to do with their lives and thinking about the impacts of their work", and some were "considering buying land somewhere because they're so scared of the instability as a result of rapid AI progress".

"They all keep this in their head on a daily basis while working on the technology."

But Coxon does have some optimism about the future of AI, telling the BBC that the people researching the technology "genuinely want to see the upside" of things like "solving diseases and improving everyone's lives" with AI.

His comments about AI have been shared millions of times on social media, and Anthropic scientist Evan Hubinger responded on X that he agreed with Coxon.

"We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade," Hubinger said.

Leading figures in the AI field have been raising the alarm about the safety threat the tech poses for years, with the heads of OpenAI, Google Deepmind and Anthropic saying as much in 2023.

These warnings have become louder in recent weeks, as evidence emerges that firms may be struggling to control the technology.

Anthropic boss Dario Amodei calls for AI development to slow down

Why some experts increasingly fear AI will take over

The contradiction at the heart of the trillion-dollar AI race