New Shared robots.txt AI Training Opt-Out: What Site Owners Should Do

If you run a business website, a blog or a news site, you have probably wondered whether your pages are being used to train AI models, and whether you can say no. Until now, the answer involved keeping track of a different crawler name for every AI company. That is starting to change. This article is for website owners, marketers and the developers or agencies who look after company sites.

Key takeaways

• On 30 October 2026, Google, Microsoft, OpenAI and Anthropic announced support for a shared way to state AI training preferences in a site’s robots.txt file.
• The approach comes from the IETF’s AI Preferences (AIPREF) work and uses a single token, __usage_ai-training, instead of one rule per company.
• Support is being phased in. Reporting says Anthropic already uses it, OpenAI plans to implement it in the coming months, Microsoft will honour training opt-outs from February 2027, and Google plans support in 2027.
• The signal is voluntary. It guides well-behaved crawlers but does not technically block anyone.
• Keep your existing company-specific rules in place for now.

What has changed

robots.txt is a plain text file at the root of a website (for example, yoursite.com/robots.txt) that tells automated crawlers which parts of the site they may visit. For years, AI companies have each published their own crawler names, such as GPTBot for OpenAI, ClaudeBot for Anthropic and the Google-Extended token for Google. If you wanted to opt out of AI training, you had to list each one separately and keep up as new ones appeared.

The new shared signal lets you write one rule that participating companies agree to read as your AI training preference. It is tied to an IETF draft standard, which means the technical details are being worked out in an open standards process rather than by a single vendor.

What a rule looks like

Coywolf, which reported on the announcement, gives an example along these lines: a User-Agent line naming __usage_ai-training, followed by Disallow: / to opt the whole site out of training, and an Allow line for a folder you are happy to share, such as a press kit. Because the standard is still a draft, the exact syntax may change, so check each provider’s documentation before you rely on a copied example.

Training is not the same as search or AI answers

An important detail is that the shared signal is about training, meaning whether your content can be used to build future AI models. It does not clearly decide whether an AI assistant can read your page to answer a user’s question in real time, which is often called grounding or retrieval. Ordinary search indexing may also follow separate rules.

Microsoft, for example, describes separate AI training and AI grounding preferences. Google says that if a site uses the new token to opt out of training without a separate Google-Extended rule, it will treat the site as having disallowed Google-Extended.

For a business, this distinction matters. Many companies are happy for AI assistants to recommend their products or quote their opening hours, while preferring that their original articles, photos or research are not used as training data. You may need to set those preferences separately.

The limits you should know

• It is a request, not a lock. Crawlers that ignore robots.txt will not be stopped by it. If you need technical blocking, bot-management services from your hosting or CDN provider are a separate step.
• Not everyone supports it yet. The timelines above show that most providers will only honour it in the months ahead.
• The standard is still moving. Other approaches exist alongside it, including a proposed Content-Usage HTTP header in a related IETF draft and Cloudflare’s own Content-Signal directives.
• robots.txt measures nothing. It will not tell you whether AI tools are citing you or sending you traffic. You will need your analytics for that.

Why this matters for African businesses and publishers

Nigerian and African publishers, schools, agencies and small businesses produce a growing amount of local-language and locally focused content that is valuable to AI developers. A simple, shared signal lowers the technical barrier to expressing a choice about how that content is used. At the same time, being visible in AI-powered search and assistants is becoming a real source of customers. The goal is not to block everything, but to make a deliberate decision rather than relying on whatever your web developer set up years ago.

Next steps

1. Look at your current file. Open yoursite.com/robots.txt in a browser and note any rules for GPTBot, ClaudeBot, Google-Extended or other AI crawlers. Ask whether each one was a deliberate decision.
2. Agree a policy with whoever owns your content: are you comfortable with AI training, with AI answers that cite you, and with ordinary search? Write the three answers down separately.
3. Keep existing rules for now. Do not remove company-specific crawler rules until each provider confirms it honours the shared token.
4. Plan to add the shared token once you have checked current provider documentation for the final syntax, and test changes on a staging site if you have one.
5. Record changes and watch results. Note the date of every robots.txt change and monitor search and referral traffic afterwards.
6. Talk to your host or developer if you need stronger protection than robots.txt can provide.

Sources

• Google, OpenAI, Microsoft, and Anthropic back a shared robots.txt AI training signal (Coywolf)
• Google, Microsoft, OpenAI and Anthropic Back a Shared AI Training Opt-Out Signal (DEV Community)
• Everything That Happened in AI Today, Monday November 2 (The Neuron)

Featured image: Photo: “Globe Computer Code and Security” by perspec_photo88, CC BY-SA 2.0, via Openverse — source