On October 30, 2026, Google, Microsoft, Anthropic, and OpenAI announced support for a shared signal that would let website owners say, with a single robots.txt entry, that they don’t want their content used to train generative AI [1]. Today a site that wants to say no has to set separate controls for each company: GPTBot for OpenAI, ClaudeBot for Anthropic, the Google-Extended token for Google, and so on. The proposal was written by Kevin Kelley of Anthropic and submitted to the Internet Engineering Task Force [2]. It’s an individual draft, not an adopted standard, and it expires in May 2027 unless renewed [2][3]. It also expresses a preference about how content is used. It doesn’t block crawlers [4].
This is a real improvement. Creators should have meaningful control over their work, and one clear signal beats a patchwork of company-specific rules. I’ll likely use it on my own sites. The harder question is whether protections introduced today could end up reinforcing advantages built before those protections existed.
Because safeguards that arrive after the fact can’t undo what happened before them.
The timing matters for two reasons. The earliest training datasets were assembled before meaningful opt-outs existed. And even once opt-outs arrived, they didn’t always work the way publishers expected.
OpenAI and Google introduced their company-specific controls in 2023: OpenAI’s GPTBot rule in August and Google-Extended in September [5][6]. By then, years of web content had already gone into training. Those controls don’t, on their own, undo completed training or guarantee deletion of material already collected [7].
They also weren’t always as complete as they looked. In May 2025, Eli Collins, a Google DeepMind vice president, acknowledged in court that opting out of DeepMind’s training didn’t stop Google’s search division from using the same content for its own AI products [8][9]. An internal document from that case showed Google had filtered out 80 billion of 160 billion training tokens after publisher opt-outs, yet that content could still be used by the search team [8].
And some promised fixes didn’t arrive. In May 2024, OpenAI announced Media Manager, a tool meant to let creators control how their work is used in AI training, with a goal of having it in place by 2025 [5]. By January 2025, it had missed that target [10], and as of May 2026 it still hadn’t launched [21].
The new signal is voluntary and forward-looking. Anthropic has started implementing it [11]. OpenAI aims to follow in the coming months, Microsoft is targeting February 1, 2027, and Google says 2027 [4]. Meta has endorsed the idea without committing to honor it [1].
Here’s the risk I see. If simpler opt-outs significantly shrink the pool of training material, companies building new datasets may face constraints that early entrants largely avoided. The incumbents already have trained models, and some have paid for premium content. OpenAI has licensing deals with Axel Springer and News Corp that cover training [12][13], and Google reportedly pays Reddit about $60 million a year to train on its content [14]. Many smaller developers and university labs can’t afford comparable deals. The size of that effect remains uncertain, but it’s the question worth watching.
Robots.txt is also a signal, not an access barrier. In 2024, a content-licensing firm warned publishers that multiple AI companies were bypassing it [15]. Its legal weight depends on where you are. In the EU, copyright law lets general text and data mining proceed unless rightsholders reserve their rights in a machine-readable way, and the AI Act requires general-purpose AI providers to respect those reservations [16][22]. Scientific research falls under a separate exception [17][22]. Under the new draft, a site without the entry expresses no preference at all [3]. That isn’t the same as legal permission, but it does leave the burden of objecting on website owners.
Regulators are paying closer attention to this dynamic. Antitrust lawyers note that regulators now treat concentration in AI inputs, including training data, as seriously as concentration in AI products [18]. In December 2025, the European Commission opened a formal investigation into whether Google used publisher and YouTube content for its AI features on unfair terms, and whether that disadvantaged rival AI developers [19][20].
None of this makes the proposal a bad idea. The principle I’d hold onto is this: creators deserve meaningful control over their work, and the competition challenge is protecting that control without letting earlier advantages become permanent.
In leadership terms, it’s an old pattern. Once you’ve climbed the ladder, it’s easy to start talking about ladder safety. Ladder safety still matters. So does remembering who got to climb before the rules changed.
Can stronger protections for creators be put in place without locking in the companies that got there first? I think that’s the real test of this proposal. What do you think?