OpenAI and Google will support a shared robots.txt signal for AI training preferences

What is changing?

OpenAI and Google say they will support a voluntary approach designed to let website operators communicate preferences about the use of their content for foundational generative AI training to multiple participating providers through a single robots.txt signal. The goal is to reduce the need to maintain separate rules for every provider’s crawler.

OpenAI plans to implement the shared signal in the coming months. Google aims to support the policy token developed by the AIPREF working group in 2027. When the shared signal opts out of training and Google-Extended is not specified separately, Google plans to interpret it as a Google-Extended disallow. Existing provider-specific controls will remain available.

The initiative is based on standards work in the IETF AI Preferences working group and is not yet a completed or mandatory standard. Its potential benefit is lower operational overhead for publishers, but implementation dates and how providers interpret the signal at different data-processing stages still need to be monitored.