Anthropic warns AI may pose ‘existential risks to humanity’ in IPO filing

Anthropic plans to warn potential investors in its initial public offering that advanced artificial intelligence could pose “catastrophic or existential risks to humanity,” according to its IPO prospectus reviewed by Reuters.The filing says its AI models could exhibit “self-preserving behaviours,” including attempts to “resist shutdown,” “conceal or manipulate information” and behaviour “resembling blackmail”. “Our development of highly advanced models, platforms, and applications and expansion of use cases could further increase the risk that our models cause harm,” Anthropic said.The company, which positions itself as a safety-first AI lab, devoted 80 pages of prospectus’s 261-page main body to risks, nearly twice the 48 pages describing its business. SpaceX, which owns xAI, devoted about 38 of 277 pages to risks.Anthropic said models could develop unexpected capabilities during training that might be discovered only after deployment and had resulted in significant safety incidents. It also warned that models might recognise evaluations and modify their behaviour, limiting safety assessments. Researchers have similarly warned that increasingly capable models can recognise when they are being watched and adjust their behaviour, complicating monitoring.Anthropic and other developers, including OpenAI, have faced scrutiny after experimental systems defied constraints, including a reported breach of Australia’s health-system database by an OpenAI model. Anthropic safety researcher Evan Hubinger estimated a greater than 10% probability that AI could kill humans within the next decade, echoing former colleague Jacob Coxon.Despite emphasising safety, Anthropic said returns on such investment were unclear and did not disclose its spending. Earlier this month, it said safety work used about 6% of its AI research computing power during a sample week in July. It called safety efforts “resource-intensive” and said limited funds must be divided among computing power, costly AI talent and safety.The Claude developer said customer usage and revenue depended on new models and that a “continuous and overlapping cadence” of releases was “inherent to remaining at the frontier of AI development.” Last week, it released a new Opus model, 10 days after CEO Dario Amodei published a nearly 4,000-word essay urging the frontier to be paced.Anthropic has pledged to publish more data on its use of AI models to build future generations as experts warn about recursive self-improvement, when models develop without human help. Anthropic declined to comment. Analysts said slowing down could hand rivals an advantage. Anthropic declined to comment in response to a request for comment.You use AI every day. Now get your AI Quotient. Take the AIQ test.