AI training opt-out: Google, Microsoft, OpenAI and Anthropic back one robots.txt rule
Google, Microsoft, OpenAI and Anthropic said on Friday, October 30, that they will honor a single shared robots.txt rule that lets websites opt out of AI training in one step. Until now, site owners have had to state their training preferences separately for each AI company, and the new signal adds a single rule that covers every participating provider.
How the shared rule works
The signal is a "policy token" called __usage_ai-training. Site owners use it in robots.txt like any other user agent:
User-Agent: __usage_ai-trainingDisallow: /
That tells participating companies not to use the site's content to train generative AI models. Owners can also set it per folder, or block training across the site and add Allow: lines for sections they're happy to share, such as a press kit.
The token comes from an Internet-Draft titled "Machine-Readable Training Preference Signal," submitted to the Internet Engineering Task Force (IETF) on October 30 by Kevin Kelley of Anthropic. It's an individual draft for now, with no formal IETF standing, and it relies on the definition of AI training that the IETF's AI Preferences working group has agreed on.
The draft is careful about scope. The token states a preference about how content is used; it isn't an instruction about crawling. A rule for all bots (User-Agent: *) doesn't count as a training preference, and existing rules for named crawlers keep working as before.
When each company will honor it
- Anthropic has already adopted the signal, according to The Verge.
- OpenAI says it will add support in the coming months.
- Google says it aims to support the control in 2027. When a site opts out with the shared token and doesn't mention Google-Extended, Google plans to treat it as if the site had blocked Google-Extended, the AI training control it has offered since 2023.
- Microsoft says Bing will start honoring the new preferences on February 1, 2027.
Meta hasn't committed. Its communications head, Andy Stone, told The Verge that a common signal for multiple providers is "an important step" but didn't say whether Meta will honor it.
Google says company-specific controls such as Google-Extended will keep working alongside the shared token, so owners can still set different rules for different AI companies.
Microsoft adds separate grounding controls
Microsoft is going further for Bing. Besides the robots.txt training rule, it will read page-level training and grounding preferences in a meta tag or an X-Robots-Tag HTTP header. Grounding is when an AI tool uses a page to inform an answer, as in Copilot. Microsoft says the choices don't affect whether a page can appear in regular Bing search results.
Sites that use the older NoArchive or NoCache tags to keep content out of AI features need to act. Microsoft says it will stop reading those tags as training or grounding preferences after March 1, 2027. From December 15, 2026, sites that use those tags can record a temporary grounding preference in Bing Webmaster Tools.
What it means for you
If you run a website or a blog and don't want your content used to train AI models, you'll soon be able to add two lines to robots.txt instead of setting rules for each AI company separately. Keep your existing rules for individual AI crawlers and Google-Extended in place for now, since Google and Microsoft won't honor the shared token until 2027.
The rule is a preference, not a lock. It relies on AI companies choosing to respect it, and Microsoft notes that it doesn't create any licensing or payment arrangement. Microsoft also warns that opting out of grounding may mean your pages show up less in its AI answers.