Anthropic’s AI oversight isn’t democratic enough

Anthropic recently announced the onboarding of six external evaluators from the nonprofit Model Evaluation and Threat Research to provide external evaluation of AI models throughout the development process. This is an important advance in our society’s ability to manage AI development.

It is also ridiculous. The current implementation is akin to a student choosing which subjects to be tested on and then choosing which of their friends to test them on that subject.

External evaluators should be embedded at AI companies, but they should not be restricted to AI safety researchers, and they should not be selected by the company being evaluated. Rather, they should represent communities across society, address a much broader set of anticipated disruptions and be selected by the public.

Were AI developing more slowly, its development could be managed by the usual democratic processes. But the incredible pace at which these technologies are advancing requires new ways for the public to manage this industry. It requires the development of new infrastructures of democracy. And independent embedded evaluations should be part of that new infrastructure.

I am a mathematician studying the internal geometry of AI systems. For years, I have been concerned that the way AI researchers are developing AI is growing untethered from the needs and desires of the public.

Two years ago, I co-organized a workshop at NeurIPS — one of the largest annual AI conferences — to facilitate dialogue between AI researchers and communities being impacted because it is important to have more public input into how AI systems are designed and developed. What is most scandalous about AI researchers is not the potential worst-case scenarios we are willing to tolerate, but the potential best-case scenarios we are planning for.

Some researchers think that AI will lead to human extinction. A much larger number believe that AI will lead to an unprecedented concentration of power and capital — that the systems we are developing will undermine the dignity of workers, including ourselves, and will empower ever more suffocating networks of surveillance. We need to mitigate all of these risks, not just the most extreme.

But our society will be unable to address any of these risks as long as the public is relegated to shouting in from the outside. In order to bring democratic oversight to the development of AI, representatives looking out for the public’s interests should be embedded in all teams building advanced AI.

Each frontier AI lab should commit “ongoing, employee-like access to a team of embedded third-party evaluators,” as Anthropic CEO Dario Amodei envisions . These evaluators would have full visibility on the development process, including “training pipelines and processes,” and be paid by and beholden to the public — not the company.

But we must go beyond Amodei’s vision. The range of issues being evaluated must be more than, “Is this system likely to cause mass casualties?” Risks from AI include damaging psychological effects such as AI psychosis , disruption to creative fields, automated killing, surveillance that expands past our constitutional rights and the erosion of our democracy itself.

Experts on all of these risks should be embedded at AI companies, not only monitoring the capabilities of new models but actively engaging with AI developers in curbing the worst effects.

For example, we have seen a horrific emergence of suicides by children that were spurred or encouraged by chatbots that the children confided in. The public deserves to have independent examiners, working alongside AI researchers at frontier labs, who have both expertise in psychology and social science as well as real-time visibility on the cutting-edge.

It is essential that outside experts work side-by-side with AI researchers during development, when adjustments can be most effectively incorporated, rather than being limited to evaluating models post-hoc when the range of available interventions is narrower. This is what AI built in a democracy looks like: A range of experts — publicly funded and accountable to the public — providing independent guidance in the development of the most consequential technology of our time.

Jacob Coxon, an AI researcher who quit his job at Anthropic over what he sees as the overall recklessness of the industry, said that it’s crucial to “keep your eyes open about the direction things are heading and not necessarily leave it all up to the people that are building it because, you know, they want engagement.”

It is time for AI researchers to put our money where our mouth is and engage with the general public. This can’t be a parody of engagement like a “contact us” button collecting unread emails. Rather, it must be a serious integration of democratically selected experts working alongside AI researchers as we create the future all of us will inhabit.

Yaim Cooper is a mathematician and deep learning expert.