A coordinated shift among the Big Four
In the last week of October 2026, a coordinated move between OpenAI, Google, Microsoft, and Anthropic signaled what many analysts consider the first concrete step toward a shared understanding: the current model of web scraping for language model training has reached a structural impasse. The four companies agreed to adopt a unified technical standard that allows website owners to opt out of having their content used for AI training through a single signal, rather than negotiating individual configurations with each crawler.
The proposed standard, formalized as an IETF draft (draft-kelley-aipref-training-preference-signal) and authored by Kevin Kelley, a standards engineer at Anthropic, introduces a "policy token" called `__usage_ai-training` that can be inserted into any website's traditional robots.txt file. The syntax requires no new directives — it leverages the existing robots.txt structure, with a group whose User-Agent is the policy token. When a crawler detects this signal, it can restrict access to resources marked as restricted.
Implementation varies across companies. Anthropic has already begun implementing the signal, according to Kelley's message to the IETF ai-control discussion list. Google and Microsoft plan to support it in 2027. OpenAI stated it will implement the signal in the coming months. Microsoft promised additional controls: publishers will be able to opt out of AI training, AI grounding, or both, while remaining in Bing search results.
Why crawlers need a new protocol
The robots.txt protocol, created in 1994, was sufficient for controlling access by conventional search engines — Googlebot, Bingbot, and similar tools. But the AI crawler ecosystem has exploded in scale: there are now estimated to be over 300 crawlers dedicated to collecting data for model training, each with its own set of policies and standards. The problem is not just the volume, but the fragmentation.
Before this unified standard, an editor wanting to block AI crawlers needed to add separate rules for each crawler — OpenAI's GPTBot, Google's CCBot, Microsoft's BingBot, Anthropic's Claude-bot, and dozens more. This generated increasingly heavy and confusing robots.txt files, with rules that conflict with each other and quickly become obsolete as new crawlers emerge.
Furthermore, AI crawlers have radically different behavior from traditional search engines. While a traditional Googlebot crawls pages for indexing — and indexing can be done with summaries and snippets — an AI crawler may need to retrieve and store the full content of every page. This fundamental distinction made the difference between "allow crawling" and "allow training" increasingly urgent, but the traditional robots.txt made no distinction between these two uses.
The policy token solves this by creating a semantic layer over robots.txt: a site can allow Google to index its articles for display in Search, while simultaneously declaring that its content should not be used to train AI models. This granularity was impossible with the old system.
The European regulatory context
Europe was already heading in this direction. The European Data Protection Board's (EDPB) Guidelines 03/2026, adopted in July 2026, establish that technical signals such as robots.txt, CAPTCHAs, and login walls must be considered relevant indicators of "reasonable expectations" of data subjects. This means a site using the new policy token will have additional regulatory backing to sustain a GDPR privacy violation claim.
The EDPB also explicitly rejects consent as a legal basis for scraping at scale, considering that data scrapers often have no direct relationship with the data subjects and cannot identify and obtain consent from each person. This leaves legitimate interest as the primary legal avenue — and the new policy token strengthens the three-part test the EDPB requires for this ground.
But the standard also faces criticism. The Electronic Privacy Information Center (EPIC) published a study in May 2026 showing that major AI companies frequently make opt-out difficult through manipulative design patterns — hidden forms, login requirements before accessing the opt-out option, and requirements for multiple separate forms. The mere existence of a unified standard does not guarantee it will be implemented correctly.
The economic argument
From an economic perspective, the standard reflects a shift in the power dynamic between AI infrastructure providers and content creators. For years, the dominant assumption in the industry was that the public web is a commons available for extraction. But as the scarcity of high-quality training data becomes a real bottleneck for model training, the ability of publishers to control data flow is becoming a valuable asset.
OpenAI itself implicitly acknowledged this point when it stated that, with the new standard, "publishers can express their AI training preferences in a unified way, without needing to configure each platform individually." This language explicitly recognizes that the value of data control lies in network effects — a single standard only works when the largest platforms respect it.
Microsoft went further by offering differentiated controls: in addition to the training opt-out, publishers will be able to exclude "grounding" (the use of content as reference for model-generated responses) while remaining in search results. This distinction is economically significant: grounding is more intrusive than simple indexing, as the model stores and references specific excerpts from the content, increasing the risk of traffic substitution from the original site.
What comes next
Practical implementation of this standard will be the real test. Robots.txt has always been a goodwill-based protocol — there is no technical enforcement mechanism. A crawler simply needs to ignore the file to violate the rules. The difference now is that the four largest AI companies in the world have publicly committed to respecting the new signal, which increases reputational and potential legal consequences for those who ignore it.
But the history of crawlers shows that AI companies often modify their practices as their data needs grow. When OpenAI needed more training capacity, when Anthropic expanded its datasets, when Google needed more data for Gemini — each expansion found justifications for bypassing existing restrictions. The policy token is different because it is an open technical standard, not a unilateral corporate policy. If it survives the pressure of scale, it will be a milestone. If not, it will be just another well-intentioned protocol forgotten when the thirst for data becomes unbearable.
Sources: The Verge, IETF, Search Engine Roundtable
✓ Independent sources cross-checked and verified before publishing
