Anthropic has told California Governor Gavin Newsom that government officials should be able to ask a court to stop a risky AI model before it reaches the public, according to Reuters. The letter is dated October 15, 2026. Its core line is blunt: “When a review finds a model poses a serious risk, the government should be able to ask a court to block it.”
The proposal comes from a company that is also preparing to list on Nasdaq, which makes the timing hard to ignore. We covered the filing in our report on Anthropic’s IPO filing and the four models that escaped tests, and the safety questions raised there sit right behind this letter. Here is what the Anthropic Newsom letter says, what it leaves open, and why a frontier lab would lobby for tools that could be turned against its own products.
One caution up front. Nothing in the reporting shows that this idea is law, a bill, or enacted California policy. No court has issued such an order, and no specific AI model has been named as a target. This is a company’s recommendation to a governor, nothing more yet.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
What Anthropic Told Newsom on October 15
The letter is addressed to Newsom and sets out a short list of policy asks for how California should handle the most capable AI systems. Reuters reported the main ask: officials should be able to seek a court order blocking the release of a model that poses a “serious risk” to the public. That wording, “serious risk,” is the phrase the coverage keeps returning to.
A second Reuters account adds the trigger. If the California Attorney General determines that a model poses a “near-term risk of catastrophic harm,” the Attorney General should be able to go to court. So the proposal puts a single named official at the center of the process, and it routes the final call through a judge rather than through an agency order.
The letter bundles several other supports. Anthropic backs third-party safety evaluators who work inside frontier AI companies and report what they find to the state Attorney General. It wants large developers to publish risk reports at least every six months. And it supports companies holding technology that can quickly shut down models, whether those models are offered to users or used internally. Reuters carried both the evaluator and kill-switch points.
The Court-Order Proposal in Plain Terms
Strip away the policy language and the idea is simple. A government lawyer would ask a judge to stop an AI model from being released, based on a review that found serious danger. The judge, not the lawyer and not the company, would decide whether the block holds.
That structure matters. Many AI safety proposals rely on companies policing themselves, with regulators reading the reports afterward. Anthropic’s version adds a hard stop that sits outside the company. It also keeps the stop under judicial supervision, which gives developers a forum to argue back, and that detail is probably part of why a lab can support it.
The proposal says “the government should be able to,” not “the government must.” It describes a power, not a duty. Nothing in the reporting says how often officials would be expected to use it, who would write the review standard, or how fast a court would need to act.
The “Near-Term Risk of Catastrophic Harm” Threshold
The threshold language is the most consequential piece. “Near-term risk of catastrophic harm” is a high bar. It does not cover bias, copyright disputes, or ordinary misuse. It points at the narrow band of scenarios that frontier labs themselves describe as the worst case.
The coverage does not define “catastrophic,” and that gap is where the fight will happen. A definition tied to mass-casualty misuse would cover few models. A looser one could reach a broad set of products. Because the reporting gives no definition, any claim about how many models the test would catch is guesswork, and we are not offering a number.
The word “near-term” also does work. It ties the trigger to risks that could materialize soon, not to long-horizon scenarios. That keeps the tool aimed at models that are close to release or already running, which fits the court-order mechanism better than a speculative standard would.
What a Court Order Could Actually Stop
Reuters reported that an order could suspend a model’s deployment, its internal use, or its further development. Those are three very different levers, and the second and third are the ones that surprised readers.
Deployment
Blocking deployment is the familiar case. A model sits ready, the review finds serious risk, and the court says it cannot ship. This mirrors how regulators treat drugs or aircraft before launch, and it is the easiest version of the idea to defend.
Internal use and further development
Internal use is where the proposal gets aggressive. Frontier labs run their strongest models on their own work, including coding and research, long before the public sees them. A court order reaching that internal use would put a judge inside the development cycle. Further development goes one step beyond, because it would let a court pause the training or refinement of a model itself.
It is unusual for a company to endorse that reach for its own sector. It tells you how seriously Anthropic wants the proposal read, and it hints that the firm sees internal deployment as a real risk surface. That concern lines up with the agent-safety issues we flagged in our look at Claude Code v2.1.311 letting agents run rm without approval.
Independent Evaluators Inside the Labs
Anthropic also supports third-party safety evaluators who work inside frontier AI companies and report identified risks to the state Attorney General, per Reuters. The phrase “inside” carries the weight. These evaluators would not test a finished product from the outside. They would sit in the company, see work in progress, and answer to a state official instead of an employer.
That design solves a real problem. Outside red-teamers often get access only to a polished release, after the key decisions are made. Embedded evaluators could catch trouble earlier. The tradeoff is trust and secrecy. Labs guard model weights and training details closely, and letting a third party in with a reporting duty to the state raises hard questions about confidentiality, liability, and who pays.
The coverage does not say who would certify these evaluators, how they would be chosen, or what happens when a lab disputes a finding. Those details decide whether the idea is a real check or a paperwork exercise.
Risk Reports Every Six Months
The letter says large AI developers should publish risk reports at least every six months. Reuters carried the cadence. Two reports a year is a clear, checkable rhythm, and it is far more frequent than the annual or one-off disclosures that many companies publish today.
The word “publish” is doing work as well. A public report invites outside scrutiny from researchers, journalists, and rival labs. It also creates a paper trail. If a company reports a manageable risk in spring and a court later hears evidence of a serious one, the earlier report becomes evidence.
What the reporting leaves out is the definition of a “large” developer. The size line would decide which companies carry the burden, and it would shape how much the rule costs startups that are still scaling.
Kill Switches: The Shutdown Capability
Anthropic says it supports AI companies having technology that can quickly shut down models offered to users or used internally. Reuters carried the point. The idea pairs naturally with the court-order mechanism. A judge’s block means little if a company cannot actually stop a running system fast.
The engineering is harder than the phrase suggests. Shutting down a chat product is easy. Shutting down an autonomous agent that holds credentials across many systems, or a model already copied into customer environments, is not. The letter, as reported, does not specify a response time or a technical standard, so “quickly” stays undefined.
Confirmed vs. Unconfirmed: A Fact Table
Because this story moves fast and invites exaggeration, the table below separates what the reporting supports from what it does not.
| Item | Status | What the reporting says |
|---|---|---|
| Letter to Gov. Newsom dated Oct. 15, 2026 | Confirmed | Anthropic sent it, per Reuters |
| Court order for a model posing “serious risk” | Confirmed (proposal) | Government should be able to ask a court to block release |
| Attorney General trigger | Confirmed (proposal) | “Near-term risk of catastrophic harm” finding |
| Order scope | Confirmed (proposal) | Deployment, internal use, or further development |
| Risk reports | Confirmed (proposal) | At least every six months for large developers |
| Any court has issued such an order | Unconfirmed | No reporting of any order |
| Proposal is law or a bill | Unconfirmed | It is a recommendation, not enacted policy |
| A specific AI model targeted | Unconfirmed | No model is named |
Why a Frontier Lab Asks for Its Own Oversight
It looks odd for a company to lobby for tools that could be aimed at its own products. Three motives are plausible, and none is confirmed by the reporting, so treat them as analysis, not fact.
First, credibility. A lab that asks for a hard backstop signals confidence in its own safety work and sets a bar competitors must match. Second, predictability. Companies often prefer a defined legal process, with a judge and an appeal path, to an unpredictable patchwork of agency actions or emergency orders. Third, public trust. Anthropic’s public identity leans heavily on safety, and a letter like this reinforces it ahead of a major capital-markets moment.
The skeptical read is that a court-based, case-by-case regime may be lighter than the prescriptive rules some lawmakers want. A judge-gated process with high thresholds could, in practice, rarely fire. Critics of any lab-written policy will ask exactly that question. The letter does not settle it.
Market Impact: The IPO Backdrop
Investors will read this letter alongside the S-1. Our earlier coverage of the filing put sales at $16.3B against a loss of $80.7B, a gap detailed in our breakdown of Anthropic’s IPO filing. A company with that profile faces regulatory risk as a core line item, and a court-order regime is a new kind of risk to price.
If the idea were adopted, a block on a flagship model could freeze revenue tied to it. A pause on internal development could slow the next release. Those are real exposures for any lab, and public-market buyers would want them disclosed. At the same time, a clear legal process could reduce the odds of sudden, chaotic intervention, which investors tend to dislike more than defined rules.
The effect on rivals depends on whether they follow. If other large developers accept similar terms, the cost spreads evenly. If only Anthropic does, it could carry extra compliance work while competitors move faster. The reporting does not mention any response from OpenAI or Google, so we make no claim about their positions.
How the Proposal Compares With Other Approaches
Governments are testing several tools at once. The table compares the general mechanisms, not specific statutes, so it avoids claims the reporting does not support.
| Approach | Who decides | Can it stop a model? | Main weakness |
|---|---|---|---|
| Voluntary company commitments | The company | Only if the company chooses | No outside enforcement |
| Published risk reports | The company, read by the public | No | Disclosure without a remedy |
| Anthropic’s court-order proposal | Attorney General asks, a judge rules | Yes, deployment, internal use, or development | Vague threshold, untested process |
| Embedded third-party evaluators | Evaluator reports to the state | Indirectly, through a finding | Confidentiality and certification |
| Government usage bans | A ministry or agency | For public-sector use only | Does not touch private deployment |
The last row has a live example. Our report that Sweden’s government blocked ChatGPT, Claude and Copilot on Oct. 26 shows a different style of control, aimed at official use rather than at a model’s existence. Anthropic’s proposal targets the model itself, which is a much wider claim of authority.
Historical Context: California and Frontier AI Rules
California has been a testing ground for AI regulation for years, partly because so many frontier labs sit inside its borders. Earlier efforts to regulate the largest models have drawn heavy industry pushback, and the governor has had to weigh those trade-offs. The reporting on this letter does not describe any specific bill now pending, so we will not guess at one.
The broader pattern is that labs have moved from opposing oversight to shaping it. Writing the terms yourself beats having them written for you. The same dynamic shows up in money, too. Our look at AI super PACs clashing in the 2026 primaries found one camp backed by $125M and another by $100M or more, which tells you how much both sides are spending to steer the rules.
Safety concerns are also moving beyond software. The UK’s plan, covered in our piece on the UK AI biosecurity lab and its up-to-£200M credits, ties AI risk to biological misuse, one of the scenarios most often cited when people talk about “catastrophic harm.”
Open Questions and Risks
Several hard issues sit under the headline. Due process comes first. A company facing a block on internal development would need a fair chance to contest the evidence, and courts are not built to evaluate cutting-edge model behavior at speed. Judges would lean on expert testimony, and the quality of that testimony would drive outcomes.
Jurisdiction comes second. California can reach companies operating within the state, but models are trained and served globally. A California order against a model may not stop it running elsewhere, and a company could simply move work. Federal law could also pre-empt or complicate any state-level approach, though the reporting does not address that.
Third, there is the risk of political misuse. Putting a power to block products in one elected official’s office invites pressure, whether from industry, advocates, or campaigns. Judicial review is the safeguard, but the Attorney General’s discretion to file would still shape which models face scrutiny.
Last, the proposal says little about open-weight models. Once weights are public, no court order can recall them. Whether the idea covers only closed, hosted models is not answered in the coverage, and it is a question open-source developers will press hard.
Five Predictions
These are our forecasts, labeled as such. None comes from the reporting.
- Rivals will respond within weeks. Expect at least one other large lab to publish its own position on court-ordered blocks, either endorsing a version or arguing for narrower terms.
- The threshold language will get sharper. If any bill picks up the idea, “catastrophic harm” will gain a written definition, because vagueness will draw legal challenges.
- Six-month reports will spread first. Of the proposed pieces, a reporting cadence is the cheapest to adopt and the easiest to voluntarily match.
- Open-weight developers will push back. A court-block regime that cannot reach published weights will look lopsided to them, and they will say so.
- Investors will ask about it. Expect analyst questions on regulatory exposure around Anthropic’s listing, with the letter cited as evidence both of maturity and of risk.
What Developers and Enterprise Buyers Should Do Now
Nothing changes legally today, so the practical advice is about readiness. If you build on a frontier API, ask your vendor how it would handle a sudden suspension of a model version, and keep a fallback provider tested and ready. Our guide to function calling across four LLMs shows how to keep your code portable between models, which is exactly the hedge that a model-level block would reward.
Teams running agents should review their own shutdown paths. If you cannot stop an agent fast, a vendor kill switch will not help you. Track the story through our AI and machine learning coverage, where we will update as reporting firms up.
Frequently Asked Questions
What did Anthropic ask Gavin Newsom to do?
In a letter dated October 15, 2026, Anthropic said government officials should be able to seek a court order blocking an AI model when it poses a “serious risk” to the public, according to Reuters.
Who would ask the court to block a model?
Per Reuters, the California Attorney General would be able to seek the order if the office determines a model poses a “near-term risk of catastrophic harm.”
What could a court order stop?
The reported scope covers suspension of a model’s deployment, its internal use, or its further development.
Is this proposal already law in California?
No. Nothing in the reporting shows it is a bill or enacted policy. It is a recommendation from Anthropic to the governor.
Has any court actually blocked an AI model?
No such order has been reported, and no specific AI model has been named as a target.
What else does the letter support?
It backs third-party safety evaluators working inside frontier AI companies and reporting risks to the Attorney General, published risk reports at least every six months from large developers, and technology that can quickly shut down models offered to users or used internally.
Why would an AI company ask for tools that could be used against it?
The reporting does not state Anthropic’s motive. Plausible readings include building credibility, preferring a defined legal process to unpredictable intervention, and reinforcing its safety identity ahead of its IPO. Those are interpretations, not confirmed reasons.
