Open one door, close another

The artificial intelligence industry has taken a step that, at first glance, seems counterintuitive: the biggest companies in the sector are agreeing to make it easier to block their own access to web data. OpenAI, Google, Microsoft, and Anthropic announced this week that they will adopt a shared "policy token" — a signal proposed by the Internet Engineering Task Force (IETF) — that will allow any website to publish, with a single entry in its robots.txt file, a refusal to have its content used to train generative AI models.

The mechanism is simple in appearance but profound in implications. Since the web existed, crawlers have followed robots.txt instructions — that standard file created in 1994 by then-student Lou von der Embde as an exclusion protocol. In recent years, this protocol has evolved to regulate not only search indexing but the large-scale collection of content to train models that, ironically, often use the very web they scrape to learn — including content from sites that explicitly do not want machines to read them.

Before this announcement, the situation was a tangled mess of fragmented efforts. An editor who wanted to protect themselves needed to add a separate entry for each AI provider: Google's block didn't apply to Anthropic, which didn't read OpenAI's directives, and so on. Each company used a bot with its own identification, and sites had to manage an increasingly long list of restrictions. The practice was workable for those with dedicated technical resources and unbearable for smaller publications.

The new standard, which has already been submitted as a formal proposal to the IETF, solves this problem with a unified signaling approach: a single line in robots.txt instructs all participating crawlers to stop collecting content for training. The specification proposed by Kevin Kelley, Anthropic's technical standards lead, introduces a token that any compatible crawler can recognize — regardless of which company operates it.

What this means in practice

For publishers and content creators, the change is significant. Today, dozens of companies collect web content for AI training, many without transparency about their practices. The new token creates a technical mechanism that transforms a declarative preference into coordinated action — instead of each site having to negotiate with dozens of companies, they just need to publish the token once.

The most interesting distinction, however, is not in the general block, but in the granularity that Microsoft introduced in its announcement. Publishers will be able to use the same mechanism to distinguish between two types of collection: model training (where content is absorbed into the model and stored in its weights) and grounding (where a system uses a live copy of a page to ground a real-time response without storing it for future training). This distinction is strategic because it touches on two distinct types of risk: data provenance and copyright issues in the first case, and accuracy and information currency problems in the second.

Meta, meanwhile, has not yet committed to implementation, though it has expressed support for the concept. The company has already faced publisher litigation over the use of content for AI training — which makes its observational stance even more curious. Meta's position was the subject of intense debate in October, when controversial updates to its terms of service raised concerns about the company's ability to use third-party content to train its models.

The legal context

The initiative does not emerge from nowhere. It connects to an increasingly complex regulatory landscape. In Europe, the AI Act already establishes transparency requirements for high-risk AI models, including rules about identifying scraping practices. In the United States, fair use issues continue to be debated in courts, without a definitive ruling for the specific case of AI training. The United Kingdom, meanwhile, abandoned a mandatory opt-out proposal in March 2026.

The technical specification, as proposed by the IETF, is deliberately neutral on legal terms — it does not resolve who has the legal right to block what, nor does it decide whether scraping for training is fair use. Instead, it offers the technical layer that allows AI providers to respect publisher preferences without needing a separate agreement. It is the same principle that led the HTTPS standard to replace unencrypted browsing: the supporting infrastructure enables widespread adoption without imposing a legal position.

What to watch

The announcement raises important practical questions. Will adoption be universal? Will smaller AI startups that did not sign the declaration follow the protocol? How will compliance verification be done? And, more fundamentally, is a token in robots.txt really enough to protect a work, or will AI companies continue finding alternatives to obtain data that is not explicitly available on pages accessible to crawlers?

The most honest answer is perhaps that this is one piece of the puzzle — not the final piece, but the first widely accepted mechanism the sector needs to align interests that have been completely asymmetric until now. In a market where web content is the primary fuel for AI and, at the same time, the resource that creators most want to protect, the shared token could be the turning point for a new phase in the relationship between publishers and AI providers.

Sources: The Verge, SiliconANGLE, AI Governance Institute

✓ Independent sources cross-checked and verified before publishing