A single preference signal for publishers
OpenAI, Google, Microsoft and Anthropic said they will adopt a shared standard that lets publishers opt out of being scraped for AI training. According to Search Engine Roundtable, the control will let publishers apply their AI training preferences across multiple AI companies instead of setting them individually.
Built into robots.txt
The proposal was submitted by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). The shared "policy token" signal will live within a website's robots.txt file, which has long signaled which web crawlers can and can't scrape a site for purposes such as search indexing, archiving or AI training.
Why it matters
The use of web content as AI training data has been a long-running dispute between publishers and AI companies. Adoption of a shared standard by major players could give publishers more control over how their content is used, though how binding the standard will be and how closely companies will follow it remain unclear.
