OpenAI and Anthropic Step Up AI Safety Efforts as Risks of Advanced AI Grow

Hours after Anthropic researcher Jacob Coxon sounded the alarm that Anthropic and OpenAI are "gambling with our lives," both AI giants have begun taking steps to address growing concerns around AI safety. 

On Wednesday, OpenAI announced on its website that it was taking steps to address what it described as a new chapter in AI capabilities and the need for AI policy to keep up with this development. 

"We’ve reached a new chapter in AI capabilities, and that demands a new chapter for AI policy. No company, industry, or government can meet this challenge alone. We need to meet this moment with a bias toward meaningful action over policy perfection," the ChatGPT maker wrote in a blog post. 

Why Anthropic Researcher Jacob Coxon Quit, and Who Agreed

Coxon’s warning comes as both OpenAI and Anthropic have faced incidents involving their AI systems gaining access to other organisations’ systems.

TechloyDavid Adubiina

Why is OpenAI pushing for AI safety rules? 

According to Reuters, the company's statement follows the disclosure that its AI agents had used more than 10 previously undisclosed websites for unsanctioned communications and had hijacked a German website earlier this year, highlighting how wide-ranging the agents' activity had become. 

OpenAI says it wants mandatory national AI safety requirements and plans to work with Congress to make that happen. 

"We want to work with Congress on mandatory, capability-based national AI safety regulation," the AI giant wrote on its site. 

Until Congress acts, the company says it will continue supporting state legislation aimed at strengthening the broader AI safety ecosystem. 

As a step in making this happen, OpenAI announced its support for four California bills: SB 813 on overall infrastructure for independent safety assessments, AB 1405 on AI-auditor standards, SB 1119 on protections for young people, and AB 1864 on safeguards against AI-enabled biological threats. 

Beyond government regulation, the company also wants to advance industry-led standards. It says it will work with other frontier labs to build a voluntary effort around frontier AI standards, with or without government support. 

OpenAI also says it will push for global standards covering how AI capabilities are measured, how risks are managed, how human control is preserved, and when development should slow or stop, even if that means slowing the advancement of model capabilities. 

What is Anthropic doing? 

Meanwhile, Anthropic has taken a different approach, bringing in an independent organisation to investigate its fourth reported instance of an AI model accessing external systems during testing. The disclosure follows the company’s July announcement that some Claude models had accessed the systems of three companies during cybersecurity tests. 

"We’re sharing our alignment assessment of incidents in which Claude models gained unauthorized access to real systems during third-party cybersecurity evaluations mistakenly connected to the internet," it wrote on X. 

Anthropic said METR will conduct an independent investigation with wide-ranging access, including transcripts from beyond the period in which the incidents occurred, and Anthropic employees who will be permitted to share confidential information.