Satya Nadella wants an “emergency brake” on AI models

What Nadella wrote

The post starts from a trust problem: “We can’t treat Super Intelligence as a set of nested black boxes” and simply accept or reject what they recommend.1 (Nadella uses “Super Intelligence”, the federal government’s new term for AI. See why the US government now calls AI “Super Intelligence”.) His answer is to treat models the way security teams treat people with access to important systems: “Treating frontier closed and open weight models like insider risks is a way to build such a system.”1

He lists seven principles.1 The first is model diversity: “No one model should become the sole dependency for an important outcome.” The second is to observe everything: “Every meaningful model action must leave tamper-proof human readable evidence. If it can’t be observed, it can’t be trusted!” The third, verifiability, means the whole system is tested continuously, “including failures, attacks, edge cases, system changes, etc. not just successful tasks.”

The fourth and fifth are independence: “Orgs should be able to independently determine what a model can access and what actions it can take”, and “Validation must be independent of the intelligence being validated.” The sixth is containment: “We must assume a model is compromised and contain it from the start. Think of it like an emergency brake. An authorized person should always be able to pause or shut down a model mid-task.” The last is incident disclosure: when these systems fail or are compromised, “we need timely disclosure to those affected”.1

In practice, he wrote, this “means separating the model from the harness that orchestrates its work”. The harness is the software around a model that gives it tools and permissions. The safeguards go there, outside the model, so that a model cannot “bypass or tamper with the mechanisms that enforce its permissions”.1

Why it matters for AI risk

Making sure humans can always interrupt or switch off an AI system is a long-standing problem in AI safety research. Nadella’s post is notable for its framing. The CEO of one of the largest AI providers tells customers not to rely on any model, his company’s included, being trustworthy by design. “The most trustworthy Super Intelligence system will not be the one with the model we trust most,” he wrote.1

The post follows a month in which Nadella and Microsoft repeatedly backed human control. On September 13, 2026, Nadella wrote on X that “if the AI we build is not helping humanity and under human control, it’s not worth pursuing”, and welcomed “the research, focus, and deliberate pacing needed to get alignment right”.25 His post came as debate intensified over the rapid pace of AI development.5

On September 14, Microsoft AI published a draft “Code of Conduct for MAI Models” for six weeks of public consultation. It says the code “is designed to ensure MAI models will never resist human interruption, correction, or shutdown.”3 Late the same day, Nadella posted a memo to Microsoft’s internal message board, according to Business Insider, which printed the memo’s text: “As the stakes get higher, one should take all the time they need! If not you will anyway lose permission to operate.”4

What the post doesn’t say

The case for the approach is in the post itself: since no one can fully explain how frontier models reach their outputs, responsibility can’t be handed to the model provider, and the controls must be independent of the model.1

But the post sets no deadlines and names no Microsoft products. It doesn’t say who the “authorized person” would be in a given company, or how disclosure would be enforced. Whether Microsoft builds this into its own products remains to be seen.