Canberra is still arguing about an AI training opt-out. Google, Microsoft, OpenAI and Anthropic just backed one for robots.txt.

On 30 October, Google's John Mueller published "Supporting a shared AI training control for websites", alongside an IETF Internet-Draft setting out the mechanism. One robots.txt group would speak to every participating AI developer, instead of separate rules for Google-Extended, GPTBot, ClaudeBot and whichever bot turns up next. (Ask anyone who maintains one of those lists how often it changes.)

Search Engine Roundtable's Barry Schwartz did the count: by his tally, Google, Microsoft, OpenAI and Anthropic are all on board.

Barry Schwartz

@rustybrick

So Google, Microsoft, OpenAI and Anthropic are all onboard seroundtable.com/google-ai-trai…

28

30 Oct 2026

I think it's worth adding. It also does less than the coverage suggests, and the "one line" framing is how people will get it wrong. Paste it in the wrong spot and you either opt out of next to nothing or, in plenty of files, hand Googlebot a Disallow: /.

What it will do, and what it won't

The draft, draft-kelley-aipref-training-preference-signal-00, is by Kevin Kelley of Anthropic. It's an individual submission with no IETF endorsement (IETF, 2026). You write a "policy token", User-Agent: __usage_ai-training, where a crawler's name would go, then ordinary rules under it. The draft's example pairs Disallow: / with Allow: /press-kit/: don't train on my site, but help yourself to the press kit.

Copy it exactly: two underscores at the front. (A misspelt token fails silently: the draft says it has to match exactly, apart from case.) A User-agent: * group doesn't count either. The draft says wildcard groups "MUST NOT be interpreted as matching the policy token".

Google adds a twist. If a site opts out with the token and doesn't name Google-Extended, "Google plans to interpret that as if the publisher had disallowed Google-Extended" (Google Search Central, 2026). Google-Extended covers Gemini training and grounding in Gemini Apps and Vertex AI (Google, 2026), so once Google supports it, the token will take you out of Gemini grounding too. You can't split them: allowing Google-Extended allows both. Vanessa Fox, who built Google's Webmaster Central, said as much to someone who assumed grounding was untouched.

Vanessa Fox

@vanessafox

@ihuzaifashoukat Not with Google. Google treats it like a directive aimed at Google-Extended. vanessafox.com/complete-refer… See: developers.google.com/crawling/docs/…

0

31 Oct 2026

Microsoft keeps grounding separate, and the draft only defines training. Nobody honours the token yet. Google aims to support it "along with other AI providers in 2027". Microsoft starts on 1 February 2027 (Microsoft Bing, 2026). OpenAI says "in the coming months" (The Next Web, 2026). The draft's author wrote that "Anthropic has already started implementing support for this draft", but that "It'll take us some time to add support across our services" (IETF, 2026). Not live, whatever at least one outlet reported, but OpenAI or Anthropic could still beat Google and Microsoft to it.

Meta likes it too. Its spokesperson Andy Stone called a common signal "an important step", without saying Meta's crawlers will honour it.

Andy Stone

@andymstone

@rustybrick Meta, too, strongly support publishers' ability to express preferences about how their content is used for foundational generative AI training. We believe creating a common signal that conveys such preferences to multiple providers at once is an important step. And we look

9

30 Oct 2026

The draft calls the token "not a directive for crawling", and it won't stop a crawler fetching the page. It doesn't touch Google Search, AI Overviews or AI Mode. It isn't retroactive, because a preference comes from "the robots.txt file in effect when the asset was retrieved". And a licensing deal can beat it: "Private agreements MAY override preferences expressed via this mechanism."

It'll only reach honest crawlers, too. In a podcast interview written up by Search Engine Roundtable on 30 October, Google's Gary Illyes put the rise in bot impersonation down to AI, with a hedge: "I think it is AI" (Search Engine Roundtable, 2026). In SER's summary, he calls robots.txt a suggestion legitimate crawlers follow, and says to verify IP and reverse DNS rather than trust a Googlebot user-agent. He co-wrote RFC 9309, the robots.txt standard, so I'd listen. Start with your own logs: Your Website Has AI Visitors and You're Ignoring Them.

Nor will it settle Canberra's argument. In September the ABC reported the Attorney-General's department had put an "opt-out by default" model for AI training to rights-holder groups (ABC, 2026). Publishers have since fronted the Joint Select Committee on Artificial Intelligence, where Hachette Australia and New Zealand head Louise Stark said "an opt-out model shifts the burden onto the very people whose rights should be protected" (ABC, 2026). The committee's final report is due on 30 November.

I couldn't find anyone in Canberra who's commented on the token itself. The committee has heard the idea, though. Anthropic, whose engineer wrote the draft, told it in a submission that it wants a "narrow form of conditional approval" for training, with rights holders opting out through robots.txt. SBS replied that "Robots.txt mechanisms are routinely ignored or bypassed" (The Next Web, 2026). So while Parliament argues about opt-out, four of the biggest AI developers have backed the same switch, with dates running from "the coming months" to 2027. If Canberra lands on opt-out, this line is a ready-made way to do it. If publishers win, it's a courtesy on top of the law.

The sceptic's case is fair: a voluntary courtesy from companies that have already trained on years of the open web, arriving late and quietly conceding the opt-out model Stark is fighting. Worth noticing who drew its limits, too: the companies it restricts. I'd still add the line, with one condition. If you already block Google-Extended, it costs you nothing. If you don't, it will also take you out of Gemini grounding once Google supports it, so decide whether you want that first. Adding it doesn't endorse opt-out as policy. It's the only cross-company lever on offer until the law says otherwise.

(Usual disclosure: we use Claude every day at Webcoda, and this site's tooling is built on it. Factor that in.)

Where not to paste it

RFC 9309 defines a group as "One or more user-agent lines that are followed by one or more rules", ending at the first user-agent line after its rules, or the end of the file (RFC Editor, 2022). Consecutive user-agent lines share whatever rules follow, and since the grammar allows empty lines inside a group, a blank line won't separate them.

Pasted whole, with its Disallow straight after it, the block is fine above your existing User-agent: *. Trouble starts when the token's line touches User-agent: * with no rule between them.

# Fine: the token has its own rules User-Agent: __usage_ai-training Disallow: / User-agent: * Disallow: /wp-admin/ # Wrong: pasted under an existing User-agent: * line User-agent: * User-Agent: __usage_ai-training Disallow: / Disallow: /wp-admin/

The second layout is one group, so Disallow: / applies to every crawler without a group of its own. If your file has no Googlebot group (WordPress's default file doesn't have one), you've told Googlebot to stay out of the whole site. Same family of mistake as Cloudflare's training block catching Googlebot, which I covered in September: Cloudflare's AI training block also blocks Googlebot. That isn't a bug, and on Tuesday it becomes the default..

Paste only the user-agent line above User-agent: * and you get the mirror image: the token borrows the wildcard's rules, so you've opted /wp-admin/ out of training and, read literally, said yes to training everywhere else.

This week, then. Add the token as its own group, then check the file with a robots.txt tester that lets you pick the user agent, and pick Googlebot. Keep your Google-Extended, GPTBot and ClaudeBot rules (the others are in From Googlebot to GPTBot: The Technical Guide to AI Crawler Optimisation) until each company confirms support. Google says existing controls keep working alongside it.

If you've used NOCACHE or NOARCHIVE to limit Bing and Copilot answers, careful. Once Bing detects any of its new training or grounding preferences on a site, this token included, it "will disregard any NoArchive and NoCache" preferences left there, and after 1 March 2027 it stops reading them as AI preferences anyway. The replacement is Microsoft's page-level ai-grounding:disallow tag or header.

And if AI Overviews and AI Mode are your real worry, the token won't help. Google's Search Console toggle, live worldwide since 31 August, will, at a price: "Sites that opt out will not receive traffic or impressions from our generative AI features" (Google, 2026).

I can't tell you whether this line becomes what Canberra points to after 30 November or a footnote to the committee's report. Nobody can yet. Right now it's a set of promises, two of them stamped 2027. Add it, put it in the right place, and keep the old lines until it counts.

---