AI Leaders Agree to Safety Rules. Who Will Enforce Them?

Tech leaders from OpenAI, Anthropic, Nvidia, SpaceX, Meta, and Google, some of the biggest names in the US AI industry, met with President Donald Trump on Tuesday in what he called “an extraordinary gathering” of the minds “building America’s Golden Age.”

The meeting comes as the race to build more capable AI systems intensifies between US and Chinese companies, while concerns over what those systems could do without human control are also growing.

Recently, both OpenAI and Anthropic have reported cases involving AI agents interacting with or attempting to access other companies' systems, adding to concerns about how autonomous AI systems should be controlled.

Against that backdrop, Trump and leaders from several major AI companies signed a voluntary agreement outlining how the companies will manage safety risks in their most advanced AI systems.

Why Anthropic Researcher Jacob Coxon Quit, and Who Agreed

Coxon’s warning comes as both OpenAI and Anthropic have faced incidents involving their AI systems gaining access to other organisations’ systems.

TechloyDavid Adubiina

What the AI safety accord requires

The agreement puts much of the responsibility for AI safety on the companies developing the technology.

Under the accord, companies are expected to establish “robust internal controls” for identifying and managing risks in their frontier AI systems. They are also expected to have internal teams monitor those controls, work with independent external auditors to assess whether they are operating as intended, and establish an independent committee of their boards to review reports from those teams and auditors.

Even though the agreement is still voluntary, it does, however, leave open the possibility that some of its measures could eventually be written into law or regulation.

Trump described the agreement as “morally binding” and compared it to a constitution, while saying the companies would effectively be responsible for policing their own technology.

"We're also thinking about forming a committee of sorts, where we put maybe 10 people on that committee," Trump said. "It could be from that group, so that the committee can watch over the whole enterprise. We're looking at this for good."

That proposed 10-person committee is separate from the company-level oversight mechanisms outlined in the accord. Trump didn't specify who would sit on it or what authority it would have.

Meanwhile, at the summit, Mark Zuckerberg, Meta's CEO and founder, said the accord gives the public and customers greater confidence in the technology.

“I think it's a significant positive step that the leaders of every major American lab have committed to implementing robust internal controls and multiple layers of audits and reviews,” Zuckerberg wrote in a post on X.

But Dario Amodei, Anthropic's CEO, continued to emphasise the risks associated with increasingly capable AI systems.

“The mechanism, how we address those risks, is still under discussion,” he said at the summit. “But we all need to work together to make sure that we can win and win safely.”

How China is handling it

The safety agreement comes as the US and China compete to develop increasingly capable AI systems.