Anthropic proposes shared robots.txt signal letting sites opt out of AI training

The Next Web reports that Google, Microsoft, OpenAI and Anthropic will honour a single shared opt-out for generative AI training, expressed in the robots.txt file. The idea originates in an IETF draft authored by Kevin Kelley of Anthropic, released on a Friday, and the three other companies announced support the same day. At present a website wanting to decline has to maintain its own rules for every company's crawler.

Under the draft, a website includes a section for a new policy token on AI training and names the paths that should stay out of training data. The article says timing differs by company. Anthropic has begun work and will keep its existing crawler controls. OpenAI intends to put the signal into practice within months, with GPTBot controls continuing alongside. Google targets early 2027 and would treat the token as a Google-Extended block when that control is not mentioned. Microsoft will honour the preferences from 1 February 2027, replacing its NoArchive and NoCache tags, and is adding a page-level setting for grounding.

The piece stresses the limits of the proposal. The token expresses a preference rather than an order about crawling, a site without such a group has stated no preference, and private agreements can override it. The draft is only an individual submission, and the IETF has not endorsed it. Microsoft says the preferences do not create licensing or payment arrangements, and that blocking grounding will significantly reduce visibility in AI-generated experiences without altering Bing search placement.

Meta's communications head voiced support for a common signal without saying whether Meta would honour it. Google said the proposal fits the EU's general-purpose AI code of practice.