Microsoft’s New AI Rules Put Surprising Limits on What Its Models Can Do

Against the backdrop of growing concerns about AI safety, Microsoft has released a new Code of Conduct for its AI models, complete with absolute constraints, red lines, and explicit rules designed to keep AI under human control. 

Last week, former Anthropic and OpenAI researcher Jacob Coxon made a bombshell claim in his resignation post that sent the tech world into a frenzy. He argued that both Anthropic and OpenAI "are racing straight to self-improving superintelligence and gambling with our lives." 

Anthropic's lead alignment science lead echoed that concern, saying the risk of unaligned AI causing human extinction is over 10% within the next decade. Days later, Anthropic CEO Dario Amodei published an essay titled "We must pace the frontier," urging the AI industry to slow down. 

Amodei proposed giving third-party evaluators "permanent, employee-level access" to AI systems so they can verify safety measures and monitor alignment during training. Microsoft has also publicly backed the broader idea of pacing the frontier, including support for embedded evaluators inside AI labs. 

What the code actually says 

The document opens with a striking prediction, warning that “containing, controlling, and aligning such a powerful force is one of the greatest challenges humanity has ever faced.” Microsoft believes superintelligent AI systems could surpass human performance across most tasks within the next decade. That possibility is exactly why the company argues that AI needs firm boundaries before these systems become even more capable.