OpenAI, Google, Microsoft, Anthropic back new AI scraping opt-out standard
Major AI companies are adopting a unified technical standard letting websites set AI-training data preferences across multiple AI firms at once instead of individually.
Summary
The Verge reports, citing earlier coverage by Search Engine Roundtable, that OpenAI, Google, Microsoft and Anthropic have said they will adopt a new standard allowing publishers to opt out of AI scraping. With it, a publisher can state its AI training preferences once and have them apply across several AI companies, rather than configuring each company separately as is required now.
The shared policy token was proposed by Kevin Kelley, an Anthropic technical lead for standards, through the Internet Engineering Task Force. It sits in a site's robots.txt file, which has long told crawlers what they may and may not fetch, covering uses such as search indexing, archiving and AI training. The four companies say they will keep their AI crawlers away from a site when they encounter the token. Meta's communications head, Andy Stone, also voiced support for a common signal but did not say whether the company would honor it.
The article places the announcement against a backdrop of publishers contending with a steady flow of AI crawlers. It mentions the Really Simple Licensing standard, which lets publishers charge visiting crawlers, and Cloudflare's decision to block AI crawlers by default.
On rollout, Anthropic has already adopted the signal, while Google and Microsoft intend to follow in the coming year and OpenAI says it will do so in the coming months. Microsoft says it will offer granular controls for opting out of AI training or AI grounding, and that sites will still appear in Bing results regardless. Google had earlier introduced comparable controls that leave search visibility unaffected.