Silicon Valley Escalates Warnings About Existential Risks of AI

(Bloomberg) -- For years, some of the leading figures in Silicon Valley have warned that artificial intelligence might pose an existential threat to humanity even as they pushed to develop the technology. This week, the high-profile resignation of an Anthropic researcher brought that conversation to the mainstream.Jacob Coxon, a rank-and-file staffer, quit his job and accused both his former employers — Anthropic and OpenAI — of "gambling with our lives" by "racing toward super-intelligent AI." Coxon said the people building AI believe it could "kill us all by the end of the decade," a point later echoed online by others at the company. One Anthropic researcher said he believes there is a greater than 10% chance that AI eliminates all humans in the next decade.

Coxon's posts have been viewed more than 100 million times on X and elicited a flood of reactions from policymakers like Senator Bernie Sanders to pop singer Sheryl Crow, all calling for humans to get a grasp on AI before it asserts dominance over the human race. His resignation note was also met by a wave of support from employees at various AI companies, including several former colleagues.The viral attention for the remarks of a previously little-known staffer reflects the persistent public pessimism about AI — particularly in the US — as well as the surging anxiety inside Silicon Valley as the technology advances and behaves in less predictable ways.In recent days, executives at OpenAI and Nvidia have heralded the beginning of a new "AGI era," referring to a more powerful form of AI known as artificial general intelligence that would be able to match or exceed humans in most tasks. The concept is poorly defined and heavily debated. But within the tech industry, there's broad concern that AI is nearing the point when it can improve itself and potentially evolve beyond our control."I now believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term," said Paul Christiano, an AI safety researcher and US government adviser who was appointed to OpenAI's nonprofit board this week. "I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level."In late July, more than 1,000 staffers across all the major AI companies signed a petition calling for a mechanism to slow the pace of AI development. Several AI staffers working on safety have resigned from major labs in recent months. And OpenAI's top scientist, Jakub Pachocki, has said that "no one is prepared for the consequences of a continued rapid rise in machine intelligence."While these warnings have gained urgency, they're not entirely new. Throughout the AI boom, there have been so-called doomers in and around the leading AI labs warning about the future. Top executives like Anthropic's Dario Amodei and SpaceX's Elon Musk have openly mused about the probability of AI destroying humanity — or P (doom), in industry parlance. Musk has estimated the chance could be as high as 20%. Amodei has said there's a 25% possibility things go "really, really badly."