OpenAI, Google, Microsoft and Anthropic say they will adopt a proposed shared standard that allows publishers to opt out of AI scraping with a single signal, according to The Verge. The “policy token” would let sites apply AI training preferences across multiple companies at once, rather than setting them individually for each provider.
Kevin Kelley, a technical lead for standards at Anthropic, is proposing the signal as part of the Internet Engineering Task Force (IETF). The token would be placed in a website’s robots.txt file, which has long been used to indicate which web crawlers can access a site.
Anthropic has already adopted the signal. Google and Microsoft plan to support it in 2027, while OpenAI says it will implement the standard in the “coming months.” All four companies say they will restrict their AI crawlers from accessing sites that display the shared token.
Microsoft says it will offer controls allowing publishers to opt out of AI training or AI grounding, the process of directing models to verified information sources. The company notes that websites will still appear in Bing search results regardless of their AI training preferences. Google previously implemented similar controls that separate AI search features and training from general search visibility.
Meta communications head Andy Stone said the company believes “creating a common signal that conveys such preferences to multiple providers at once is an important step,” but did not confirm whether Meta will honor the standard.
Publishers currently must configure AI training preferences separately for each company. Some have pursued other approaches to control scraping, including the Really Simple Licensing standard, which allows publishers to charge crawlers, and Cloudflare’s default blocking of AI crawlers.