· 5 min read · Muhammad Shah
Google, Microsoft, OpenAI and Anthropic agree on one robots.txt opt-out for AI training
On 30 October 2026, Google, Microsoft, OpenAI and Anthropic announced support for a shared way for websites to say whether their content can be used to train AI models. Until now, every company had its own crawler name to block. Soon, one robots.txt rule should cover all four.
What changed
The new signal is a "policy token" in robots.txt called __usage_ai-training. It comes out of the IETF's AI Preferences (AIPREF) working group, and the Internet-Draft behind it is still being finalised. Allowing training stays the default, so you only need to add something if you want to opt out.
# Opt the whole site out of AI training
User-Agent: __usage_ai-training
Disallow: /
# Or opt out a single section
User-Agent: __usage_ai-training
Disallow: /members/When each company starts honouring it
- Anthropic: already rolling it out, the only one of the four using it today.
- OpenAI: says it will honour the signal in the coming months.
- Microsoft: from 1 February 2027, as Bing crawls.
- Google: aiming for 2027. If you use the new token and don't mention Google-Extended, Google plans to treat it as if you had disallowed Google-Extended.
Existing company-specific controls such as Google-Extended and GPTBot keep working alongside the new token, so for now you still need them if you want an opt-out to apply everywhere today.
Training is not the same as search or AI answers
This is the part most coverage skips. There are three different things a crawler can do with your content, and they are controlled separately:
- Search: indexing and ranking your pages in normal results. Microsoft says the new training preference does not change eligibility for Bing search results.
- Training: using your content to build future AI models. This is what the new token controls.
- Grounding: fetching your page live to answer a user's question, usually with a link back to you. Bing handles this with a separate, page-level preference, and warns that blocking it significantly reduces your visibility in AI experiences.
In other words, you can opt out of training and still appear in search and in AI answers that cite you. Blocking everything is a much bigger decision.
Should your website opt out?
My view: it depends on what your content is for.
- Marketing sites, portfolios and SaaS landing pages: usually allow training. You want AI assistants to know who you are and recommend you. This site allows it for exactly that reason.
- Publishers, paid courses and premium content: consider opting out of training while keeping search and grounding allowed, so you keep the traffic without giving the content away.
- Private or member areas: these shouldn't be crawlable at all. Protect them with logins, not just robots.txt.
Adding it to a Next.js site
If your robots.txt is generated by Next.js, add the token as another rule in app/robots.ts. In Laravel or any other stack, add the same lines to public/robots.txt.
// app/robots.ts
import type { MetadataRoute } from "next";
export default function robots(): MetadataRoute.Robots {
return {
rules: [
{ userAgent: "*", allow: "/", disallow: "/api/" },
// Opt out of AI training, keep search and AI answers
{ userAgent: "__usage_ai-training", disallow: "/" },
],
sitemap: "https://example.com/sitemap.xml",
};
}Remember: it's a request, not a lock
Like everything in robots.txt, the signal is voluntary. Well-behaved crawlers from the companies that signed up will respect it; scrapers that ignore robots.txt won't. If that matters for your content, pair it with bot protection at your CDN, such as Cloudflare's bot settings, and keep anything truly private behind authentication.
It's still a real step forward: one rule instead of a growing list of crawler names. If you'd like a quick review of your robots.txt, sitemap and AI settings, get in touch.
Sources
- Google Search Central Blog: AI training control (30 Oct 2026, John Mueller)
- Bing Blog: Your content, your choice: clearer preferences for training and grounding (30 Oct 2026)
- OpenAI: Helping websites express AI training preferences
- Coywolf: Google, OpenAI, Microsoft and Anthropic back a shared robots.txt AI training signal (1 Nov 2026)
- IETF Internet-Draft: AI training preference signal (AIPREF)
Keep reading
4 min read
Microsoft's 60,000-employee buyout: what it means for developers and the teams hiring them
Microsoft offered voluntary buyouts to about 60,000 staff in October 2026. What was actually announced, why it isn't a 60,000-person layoff, and what it signals for developers and companies hiring.
5 min read
Tech trends for late 2026: agents that work together, cheaper models and an agent-ready web
The shifts worth acting on this quarter: AI agents coordinating with each other, agent identity, WebMCP, falling model prices and outcome-based pricing, and what each one means for your product.