Publishers may soon get a simpler way to tell AI companies what they can and cannot use for model training. The catch is that the new control looks more like a shared instruction than an enforceable lock.
Digiday reports that Anthropic, Google, Microsoft, and OpenAI are backing a proposed AI training policy token that publishers could add to their robots.txt files. The proposal is still moving through the IETF’s AI Preferences work, so it is not an official internet standard yet.
A shared signal for AI crawlers
The proposed syntax is intentionally simple. A publisher would add User-Agent: __usage_ai-training, followed by an instruction to allow or disallow the use of its content for AI training. An allow instruction could also include exceptions, such as permitting access to a press kit.
That would replace the current need to manage separate instructions for individual crawlers. Anthropic says it is beginning adoption, OpenAI says it will honor the signal in the coming months, and Google says it aims to support it in 2027. Microsoft has also said it will support the token and give publishers a way to opt out of AI grounding, where crawlers collect information for real-time answers. Meta has expressed support for a common signal, but has not made the same explicit compliance commitment.
For publishers, the appeal is obvious. One preference could speak to several major AI companies at once. That is cleaner than maintaining a growing list of crawler rules while publishers are already trying to manage licensing, copyright disputes, bot blocking, and visibility in AI answers.
The control problem is still the real problem
The weakness is that robots.txt has always been a request, not a security system. Digiday cites ScalePost data showing that crawlers operated by the four supporting companies made roughly 1 billion requests across more than 50 publisher properties in a recent 30-day period, while only 0.15% of those requests were for robots.txt files.
That makes the token useful as a shared preference, but limited as a guarantee. It also does not address data gathered through third-party crawlers, open datasets, or other sources. And it creates a difficult trade-off: opting out of AI training may be attractive, while opting out of AI grounding could reduce a publisher’s visibility inside AI experiences.
The bigger signal is that publishers are being asked to express opposition through increasingly precise technical controls while the underlying power relationship remains unchanged. The token may become one more arrow in a publisher’s quiver. It does not answer the harder question of whether AI companies should need permission before using the content in the first place.