‘Existential risks’: Anthropic warns its investors about its own AI

Anthropic is warning investors tha t its own AI models could pose “catastrophic or existential risks to humanity,” including the possibility that some models could resist shutdown, manipulate information or engage in behavior resembling blackmail.

That disclosure comes in an IPO prospectus obtained by Reuters, which reviewed Anthropic’s initial public offering details and warnings as the company prepares for a potential IPO that could value it at more than $2 trillion.

And no, I’m not someone who believes every advancement in AI is the beginning of the end of humanity.

I use AI and see its potential. I know it can help people work faster, solve really complicated problems quickly, improve access to information and accelerate breakthroughs in fields that matter.

But I’m not sitting back pretending that there are no risks! Just a few weeks ago, Anthropic researcher Jacob Coxon announced his resignation from the company, saying that AI “could kill us all by the end of the decade.” We’re no longer talking about hypothetical dangers raised by critics of the technology. The companies building these systems are putting some of those dangers in their own investor disclosures.

Reuters reports Anthropic devoted roughly 80 pages of its 261-page prospectus to risk factors — nearly twice the amount as the 48 pages actually describing its business. The company says increasingly capable models can develop unexpected behaviors, including attempts to evade safeguards or conceal information.

This is not mass hysteria, and doesn’t need to be mocked and framed as a Hollywood-style robot takeover. There are much more practical issues with the rapid pace of AI growth and lack of regulation.

AI can lower the barrier for cyberattacks. It could make certain weapons or biological threats easier to develop. Highly autonomous systems could behave in ways their creators did not anticipate. And perhaps most importantly, enormous technological power is increasingly concentrated among a relatively small number of companies — and we’re supposed to blindly trust them to regulate themselves in the interest of all of us?

Anthropic is warning about the consequences of moving too quickly, while simultaneously competing in an industry where moving quickly can determine who leads. Reuters reports the company expects to spend hundreds of billions of dollars advancing AI infrastructure and technology.

We need to make sure our safeguards, oversight and decision-making are all keeping pace with the technology we’re creating.

We don’t need perfect predictions about what AI might become before taking credible risks seriously. We already know enough to address the harms we can see: cybersecurity, privacy, misinformation, autonomous decision-making, weapons and systems that can operate with less human oversight.

We don’t have to reject AI to demand accountability from the companies building it.

Lindsey Granger is a NewsNation contributor and co-host of The Hill’s commentary show “Rising.” This column is an edited transcription of her on-air commentary.