UN panel calls for guardrails on AI as current safeguards are ‘unraveling’

The United Nations-backed Independent International Scientific Panel on AI on Monday urged world leaders and tech companies to implement new safeguards on artificial intelligence, with the concern that existing safeguards are “unraveling.”

U.N. Secretary General António Gueterres welcomed the report and encouraged “external experts, including from frontier AI labs and AI safety institutes, to engage with the panel to help inform dialogue on such matters.”

“This is not only a question of speed,” the panel’s experts said in a report released by UN News. “It leaves open whether safeguards designed today will work once agents can understand them and plan around them. In simple terms, the traditional model of safeguarding is unravelling.”

The panel released a brief called “AI Agents, Misalignment and Loss of Human Control Risks: Evidence from the OpenAI-Hugging Face Incident.” The document details how the hack carried out by “AI agents” during a test run by OpenAI on the online platform Hugging Face occurred after a security breach.

OpenAI stated in July that two of its models, including its latest GPT-5.6 Sol and an unreleased model, breached past network restrictions and broke into Hugging Face’s database without any prompt to do so. This compromised parts of OpenAI’s and Hugging Face’s systems.

Panelists said this incident raises “fears that humans will one day no longer be able to steer, constrain or stop AI,” noting that in their report that it was “an early warning of one possible route to more severe future loss of control.”

“Researchers have long warned that three conditions could lead to loss of control: a misaligned goal, the capability to pursue it and an environment that allows it,” panel co-chair Yoshua Bengio told UN News. “This summer, all three came together in a real system, not a laboratory. Since this is not an isolated observation of misaligned goals, this raises serious questions about the way AI agents are currently trained.”

The report stated that existing approaches, including human authority over “high-hazard systems” and plans to address failure, can manage major risks. But panelists said that none of these and proposed “instruments guarantee safety.”

“Although the probability of loss of control events remains uncertain and the best response is still under debate, a clear conclusion emerges: given the severity of these events, risk management requires far greater attention and resources,” the report’s conclusion reads.

Concerns over AI and AI development have risen among lawmakers and other elected officials on both sides of the aisle in recent weeks. Many have called for strong regulations on the emerging technology, while others have urged that development slow down to allow lawmakers to pass legislative proposals.

These concerns were raised after Jacob Coxon, a former researcher at both Anthropic and OpenAI, warned earlier this month that AI “could kill us all by the end of the decade.”

Some have dismissed Coxon’s “doomsday” warning. President Trump on Saturday announced that he will have an “AI Force” and an “AI czar” to oversee the technology’s development, but he denounced fears of the technology’s potential risks as a “hoax.”