Major artificial intelligence developers including OpenAI, Google, Microsoft, and Anthropic have announced they will adopt a new standard that allows publishers to opt out of being scraped by AI, as reported by The Verge based on initial reporting from Search Engine Roundtable. This new control enables publishers to apply their AI training preferences across multiple AI companies simultaneously, rather than configuring them for each provider individually.
The Shared Policy Token
The proposed shared “policy token” signal was created by Kevin Kelley, a technical lead for standards at Anthropic, through the Internet Engineering Task Force (IETF). The signal operates inside a website’s robots.txt file, which traditionally indicates which web crawlers are permitted to scrape a site for search indexing, archiving, or training purposes. Previously, website owners had to communicate their AI training preferences individually to every separate artificial intelligence firm.
This development arrives as media outlets and publishers struggle with a continuous wave of automated AI crawlers harvesting their content to feed into large language models. Other entities have previously tried to address unauthorized scraping, such as the Really Simple Licensing (RSL) standard which permits publishers to charge visiting crawlers, alongside Cloudflare’s default blocking of AI bots.
Implementation and Company Stances
OpenAI, Google, Microsoft, and Anthropic state that their respective AI crawlers will respect the shared token and refrain from accessing sites that feature it. Andy Stone, the communications head at Meta, also expressed support for the new setting, though he did not confirm whether his company would actively honor it. “We believe creating a common signal that conveys such preferences to multiple providers at once is an important step,” Stone says, adding that Meta looks forward to collaborating with industry peers on standards and implementation.
Microsoft announced plans to provide granular controls enabling publishers to choose whether to block their content from AI training or AI grounding—a technique that directs models to verified sources to minimize hallucinations. Microsoft emphasized that websites will continue to appear in standard Bing search results regardless of these chosen preferences. Google has previously introduced comparable controls enabling sites to manage their presence in AI search features or training datasets without affecting traditional search visibility.
Anthropic has already integrated the new signal, whereas Google and Microsoft intend to provide support next year. OpenAI stated it will roll out the implementation within the coming months.
