Google, Microsoft, OpenAI and Anthropic Just Agreed on One robots.txt Signal for AI Training — Explained Honestly

Nothing here is sponsored — no affiliate links on this page. Every fact below comes from reporting of the 30 October 2026 announcements (Unite.AI, MonsterMegs, Blockchain.News) and the IETF Internet-Draft they describe. We have not tested any provider's implementation of the new signal, and where a detail comes from secondary reporting rather than a provider's own announcement, we label it as reported.

On 30 October 2026, four of the biggest names in AI — Google, Microsoft, OpenAI and Anthropic — lined up behind a single shared robots.txt signal for AI training preferences. For years, telling AI companies not to train on your website meant maintaining a growing list of crawler names, one vendor at a time. The new idea is simple: write one preference, in one place, and every participating provider reads it the same way. That is genuinely good news for publishers. It is not, however, a switch that turns AI training off everywhere tomorrow.

What actually changed

One signal instead of one rule per provider. Until now, a site owner who wanted to opt out of AI training had to identify each provider's crawlers and maintain separate rules for each — GPTBot for OpenAI, Google-Extended for Google, and so on. The shared signal replaces that patchwork with a single policy token that participating providers all agree to read the same way.

The announcement was coordinated, not solo. Google announced the shared control on its Search Central blog (John Mueller, Google Search Advocate), saying it "will help websites express their AI training preferences across multiple AI providers in a single step." OpenAI announced the same day that it will honor the shared signal. Anthropic authored the underlying draft, and Microsoft signalled support the same day. That coordination is the real news.

It is based on IETF work, not a private deal. The signal is defined in an Internet-Draft from the IETF's AI Preferences (aipref) working group, titled "Machine-Readable Training Preference Signal" (draft-kelley-aipref-training-preference-signal-00), authored by Kevin Kelley of Anthropic, dated 30 October 2026 and expiring 3 May 2027.

How the signal works

The mechanism is deliberately boring — and that is its strength. It introduces a single policy token, __usage_ai-training, that sits in the User-Agent line of an ordinary robots.txt group. There are no new directives and no new grammar: the path language, longest-match precedence and group structure of the robots.txt standard (RFC 9309) are unchanged.

The draft's own example group expresses a preference against AI training for all paths except those under /press-kit/:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Disallow expresses a preference against AI training for matching paths; Allow expresses a preference in favour. Because it reuses existing robots.txt path rules, a preference can be set per directory rather than site-wide.

Three compatibility details matter. Parsers that do not recognise the token simply ignore the group, so adding it should not change how search engines read the rest of your file. Wildcard groups do not count — a User-Agent: * group "must not be interpreted as matching the policy token," so with no __usage_ai-training group, no training preference has been expressed at all. And specific rules win — providers may treat their own tokens, such as Google-Extended, as more specific than the shared one, so an explicit Google-Extended allow keeps Gemini Apps grounding eligibility even under a broad opt-out.

What each provider actually committed — and when

The timelines are uneven, and that unevenness is the most important honest detail in this story. There is no single switch-on date.

Google — targeting support in 2027. Google says it aims to support the new AI training control in 2027. When a site uses the shared token to opt out and does not explicitly mention Google-Extended, Google plans to interpret that as if the publisher had disallowed Google-Extended — the training control Google has offered since 2023, which is not going away. Company-specific controls, Google says, "will continue to function alongside this new joint signal." Google also ties the move to regulation: the proposal is consistent with the EU Code of Practice for General Purpose AI model providers under the EU AI Act, which recognises robots.txt as a primary way for publishers to state machine-readable preferences.

OpenAI — "in the coming months." OpenAI announced on 30 October 2026 that it will honor the shared signal, describing the effort as voluntary and aimed at removing the burden of maintaining separate rules per provider. It says it aims to implement the signal in the coming months and will share another update with implementation details when it does. No firmer date has been given.

Anthropic — authored the draft; implementation reported, not confirmed here. Anthropic's Kevin Kelley authored the draft, and same-day reporting says Anthropic has already started implementing support. We label that as reported, because we have not verified it against an Anthropic announcement in the sources fetched for this article.

Microsoft — building on the shared preference, with a separate grounding setting (as reported). Reporting says Microsoft will build on the shared preference while adding a separate setting for AI grounding, and that publishers should not have to choose between Bing discoverability and training choice. That is reported detail, not a Microsoft announcement we verified directly — and the distinction matters: grounding (an assistant citing your pages) is not the same thing as training (your pages changing a model's weights).

What did NOT change

Your existing controls still work and are not replaced. OpenAI says the new signal will not replace the controls it already provides through robots.txt, which will continue to apply and be respected. Google says the same about Google-Extended. Do not delete your existing GPTBot or Google-Extended rules on the strength of today's announcement.

Search and training stay separate. Each provider's settings are independent: a webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTBot for training. Allowing a search bot can preserve search eligibility while disallowing training — and, on Google's side, opting out of training should not touch your search ranking.

A training preference does not stop crawling. The draft states plainly that the policy token "is not a directive for crawling behavior." Setting the shared token tells a provider how you want your content used; it does not, by itself, stop that provider's bots from fetching your pages. If AI crawler traffic is eating your server resources today, you still need user-agent groups for specific crawlers, or blocking and rate limiting at the server or CDN level.

It does not remove what was already collected. Updating robots.txt does not show that previously collected material has been removed from datasets or existing models. The signal governs future training preferences, as read by cooperating providers. It is not a deletion request and not a retroactive cleanup.

It is voluntary, and robots.txt is not enforcement. Participation is voluntary on both sides. Robots.txt is a preference signal, not access control or a security measure: a non-cooperating scraper can simply ignore it.

Who this helps — and who should wait

• Publishers, bloggers and small site owners tired of the patchwork: you are the intended audience. One group, one preference, read the same way by participating providers — once they implement it.
• Sites that want per-section control: the path rules let you opt out broadly and carve out exceptions, or vice versa. That granularity is genuinely useful, and it is already specified in the draft.
• Anyone hoping this fixes crawler load or removes old training data: wait, and do not confuse this signal with either problem. Crawler load needs server/CDN controls; old data needs a different conversation entirely.
• Anyone who wants a guarantee today: wait. Until Google's 2027 support, OpenAI's "coming months" implementation and the other providers' details arrive, the shared signal is a published preference format with public backing — not an enforced control you can rely on everywhere.

What to do now (practical, hedged)

1. Read the draft before you rewrite anything. The specification is public and short, and it expires on 3 May 2027 unless revised — so expect details to change before providers enforce it. You do not need to rewrite your robots.txt today.
2. Leave your existing controls in place. GPTBot, Google-Extended and your other provider rules are what actually work right now.
3. Audit what you already have. Confirm Googlebot is not accidentally blocked, note your Google-Extended and GPTBot rules, and remember that an explicit Google-Extended rule will override the shared signal for Google. Adding a __usage_ai-training group now is low risk — parsers that do not recognise it ignore it — but treat it as an early preference statement, not as protection already enforced.
4. Decide what you actually want, section by section. Opting out, allowing, and allowing only for some sections are all reasonable positions. Grounding visibility is a separate question from training.
5. Treat crawler load separately. If your logs show AI bots hammering your site, rate-limit or block at the server or CDN layer. A training preference will not slow them down.

What we cannot verify — stated plainly

• Anthropic's and Microsoft's implementation status: reported by same-day coverage, not verified here against their own announcements.
• Any single switch-on date: there is none. Google targets 2027, OpenAI says "in the coming months," and the draft expires in May 2027 unless revised.
• Whether the draft syntax is final: it is an individual Internet-Draft with no formal IETF endorsement or standing in the standards process. Google itself describes its commitment as a draft proposal.
• Enforcement against non-cooperating scrapers: there is none in this mechanism. Robots.txt expresses a preference; it does not compel compliance, and it does not remove previously collected material.
• Exact provider implementation details: OpenAI says it will share details when it implements the signal; Google's mapping to Google-Extended is the most detailed commitment published so far.

The honest summary is this: for the first time, the largest model builders have publicly agreed on one way for publishers to say "do not train on this" — and the mechanism is elegantly simple, reusing a file every site owner already has. But agreement on a format is not enforcement, the rollout stretches into 2027, and the controls that protect you today are still the old per-provider ones. Keep those, learn the new signal, and judge the providers by what they actually implement — not by today's headlines.

Sources: Unite.AI, "OpenAI to Honor Shared Robots.txt Signal for AI Training Preferences", 30 October 2026, https://www.unite.ai/openai-to-honor-shared-robots-txt-signal-for-ai-training-preferences/ ; MLex, "Google, Microsoft, OpenAI, Anthropic to offer streamlined AI training tool", 30 October 2026, https://www.mlex.com/mlex/articles/2544745/google-microsoft-openai-anthropic-to-offer-streamlined-ai-training-tool ; MonsterMegs, "What The Shared AI Training Control Means", 30 October 2026, https://monstermegs.com/blog/ai-training-control/ ; Blockchain.News, "OpenAI to honor shared robots.txt AI training signal", 30 October 2026, https://blockchain.news/news/openai-honor-shared-robots-txt-ai-training-signal ; Daily AI Brief, "OpenAI shared robots.txt AI training signal", 30 October 2026, https://dailyaibrief.com/news/openai-shared-robots-txt-ai-training-signal-uQrvMeDk .