OpenAI, Google, Microsoft and Anthropic Back One robots.txt Token for AI Training. Only Anthropic Honors It Today. Here's the Three-Line Decision for Your Site.
On October 30, OpenAI, Google, Microsoft, and Anthropic announced they will honor one shared robots.txt signal for AI training. The token is __usage_ai-training. Per press reports, Anthropic is already honoring it, Microsoft is targeting February 1, 2027, Google is targeting 2027, and OpenAI says "in the coming months."
For anyone who publishes content for a living, that is the first time one line of text could cover several labs at once. It is also an individual Internet-Draft, not a standard, so read the limits before you rely on it.
The syntax
The token is defined in an IETF individual draft titled "Machine-Readable Training Preference Signal," authored by Kevin Kelley of Anthropic and dated October 30, 2026. The draft introduces no new directives. You use ordinary robots.txt lines, with the token as the user agent:
User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/
A Disallow expresses a preference against use for AI training. An Allow expresses a preference in favor. Path matching follows RFC 9309, so you can opt out for most of the site and carve out folders. If no group names the token, no preference is expressed.
One trap: a User-Agent: * group does not match this token. A blanket wildcard rule will not opt you out, and the draft says so explicitly. You have to name the token.
What it does not do
This is where most of the hot takes will be wrong, so here are the limits from the draft and from coverage of it.
It does not control crawling. The draft calls the training signal and the crawler product tokens "independent and orthogonal." Your existing GPTBot, ClaudeBot, or Google-Extended rules stay in place and are evaluated separately, and OpenAI has said the new token will not replace its existing controls.
It does not cover search indexing or real-time retrieval. Those follow separate rules. If you want an AI assistant to cite your page when someone asks a question, blocking training does not block that, and it should not.
It is not enforcement. The token is a preference that depends on voluntary compliance, and legal consequences depend on your jurisdiction, not on the token.
And it is a draft. The IETF document is an individual submission with no formal standing, and it expires May 3, 2027 unless it is renewed or adopted.
The decision for a one-person site
Your choices are three. Opt out of training everywhere, opt out except for a folder, or say nothing.
For a blog like this one, I'd opt out of training and keep search and retrieval open. The business model is people finding and reading the posts. Being cited in answers brings readers. Being absorbed into a model's weights with no link back does not. The token lets me say that precisely, which the old per-bot rules only did bot by bot.
If you sell the content itself, such as a paid newsletter archive, a course, or a data product, the case for opting out is stronger. If your content is documentation for your own product, you may want training on it, because a model that knows your API is free support. Allow it, or at least allow the docs path.
Adding it to an Astro site
If you ship a static robots.txt, put it in public/robots.txt:
User-Agent: __usage_ai-training Disallow: / User-Agent: * Allow: / Sitemap: https://example.com/sitemap-index.xml
Check the file after deploy by opening /robots.txt in a browser. A typo in the token does nothing and tells you nothing, because compliance is silent. The draft says matching is exact apart from case, so copy it character for character, including the double underscore.
Where this could be wrong
The strongest counter-argument is that it changes very little today. Only Anthropic is reported to honor it now, Google and Microsoft are a year out by their stated timelines, and those dates come from press reports, not from the companies' own materials. Crawlers that never respected robots.txt will not start. If your goal is to keep content out of every model, this is a polite request, not a lock.
There is also a risk of false comfort. Adding the line and moving on can leave you feeling protected while retrieval bots, scrapers, and older training sets are untouched.
What I'd actually do
Add the token this weekend. It costs three lines and a deploy. Keep your existing per-bot rules. Do not block retrieval or search bots unless you have a reason. Set a reminder for February and again for mid-2027 to check whether Microsoft and Google shipped what they said, and whether the draft was adopted before it expires.
Author
Lukas
@lukcombinator