Major AI Developers Agree on New Standard for Publisher Opt-Out from Web Scraping
Leading artificial intelligence companies, including OpenAI, Google, Microsoft, and Anthropic, have agreed to adopt a new standardized 'policy token' within robots.txt files. This initiative allows digital publishers to communicate their data scraping preferences once to multiple AI crawlers simultaneously, rather than negotiating individually with every developer. The proposed standard, developed through the Internet Engineering Task Force (IETF), aims to resolve ongoing tensions between content creators and AI firms regarding unauthorized training data usage. While Microsoft plans to offer granular controls distinguishing between model training and 'AI grounding,' all participating companies have committed to respecting the signal by restricting their crawlers from accessing sites that opt out.
Topics: AI training data content licensing data privacy industry standards web scraping