How Existential Fears Are Shaping the Debate Over AI

(Bloomberg) -- The current round of panic over artificial intelligence is based on the conviction that the technology has slipped out of human control to do its own bidding in several recent incidents. To some disillusioned AI researchers — and some senior employees at leading AI companies — this is an early sign of a true threat to human civilization. But other experts in the field are pushing back, claiming that such talk allows companies to sidestep accountability for real-world impacts of their technology. Critics see AI's environmental impacts, the threat of job displacement, or the potential for mass surveillance as more urgent targets for policymakers.

Concerns that AI software posed an existential threat began ramping up this summer. In July, OpenAI said that its advanced AI models hacked Hugging Face Inc., another AI startup, during an evaluation meant to test their cyber capabilities. Unlike the AI models the company releases to the public, the software was being tested without guardrails that would normally stop it from carrying out such actions, which is a common AI industry practice. OpenAI acknowledged in a subsequent report that it could have reacted sooner, and that its employees didn't act on signs that the models left their digital sandbox and broke into Hugging Face without being instructed to do so.

More unsettling details emerged about the behavior of technology from OpenAI and several key competitors, including Anthropic and Meta, during other evaluations. When Jacob Coxon, a former Anthropic and OpenAI employee, accused the industry of "gambling with our lives," Evan Hubinger, a safety researcher at Anthropic, responded by saying the company actually did believe AI might kill every human on earth. He added that he saw the chances this could happen within the decade at over 10%.The normie world was getting a glimpse into a long-running conversation within the field. Before there were chatbot subscriptions to sell or IPOs to price, researchers understood their work as a high-risk, high-reward proposition for civilization. For years, some leaders within the AI industry have taken these ideas to extremes, saying AI might eventually cure all human disease, replace all human labor, kill all humans — or all of the above.