Skip to content
Now in ChatGPT & Claude via MCP See how it works →
Technical SEOIntermediateUpdated 2 Nov 2026

Google’s Shared AI Training Control: What Changes in robots.txt

Google will honour a shared robots.txt token for AI training. What it does, what it does not touch in Search, and what to set today.

AkshayCo-founder & CTO, Hyperlocal SEO AI · 6 min read

Technical SEO

Key takeaways

  • On 30 October 2026 Google said it will support a shared robots.txt token that tells AI providers whether your content may be used for training.
  • The proposed token is __usage_ai-training, from an IETF draft. Google’s crawler documentation says support is planned for early 2027.
  • If you opt out with the shared token and say nothing about Google-Extended, Google will treat it as a Google-Extended disallow.
  • Google-Extended does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode have their own control in Search Console.
  • For most businesses that want to be cited by AI assistants, the right move is to leave training and grounding open.

On 30 October 2026, John Mueller announced on the Google Search Central blog that Google will support a shared AI training control in robots.txt. Instead of writing one rule for every AI company, a site will be able to write one rule that every participating provider reads.

This matters more than it sounds. Many sites I audit block AI crawlers by copying long lists of user agents from forums, and half of those rules do something different from what the owner thinks. A shared token makes the choice simpler. It also makes it easier to get wrong at scale, so it is worth understanding before you touch robots.txt.

What Google announced

The control comes from work in the IETF AIPREF working group. The draft that defines it, draft-kelley-aipref-training-preference-signal, reuses ordinary robots.txt groups. You address a policy token instead of a crawler name:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

This example says: do not use my content for AI training, except the press kit. No new directives are needed, so existing robots.txt parsers can read it.

Google’s crawler documentation says Google intends to support this token in early 2027. The blog post says Google aims to support it "along with other AI providers in 2027". The draft is an individual Internet-Draft dated 30 October 2026. It is not an approved standard yet, so the exact name could still change.

How it interacts with Google-Extended

Google-Extended is the robots.txt token Google introduced in 2023. It controls whether content Google crawls can be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. It has no crawler of its own; Googlebot does the crawling and Google-Extended is only a permission.

Google’s rule for the new token is simple:

  • If you opt out of training with the shared token and do not mention Google-Extended, Google treats it as if you had disallowed Google-Extended.
  • If you do set Google-Extended explicitly, that company-specific rule still works alongside the shared one.
  • Sites that explicitly allow Google-Extended stay eligible to appear, with links, in Gemini Apps answers that are grounded in Search.

This is the part people mix up. Google’s documentation says Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal. Blocking AI training will not make you rank lower in normal results, and allowing it will not make you rank higher.

AI Overviews and AI Mode are part of Google Search, and they are governed by different controls:

  • Normal indexing controls (noindex, nosnippet, data-nosnippet, max-snippet) decide what Google can show from a page, in classic results and in AI features.
  • Google’s "Control what you share" page, updated on 21 October 2026, lists a site-wide Search generative AI control in Search Console that decides whether your site is included in AI Overviews and AI Mode.

So there are three separate questions: can Google index me, can Google show me in AI answers on Search, and can AI companies train on me. Each one has its own switch.

My recommendation for business websites

For a clinic, a hospital chain, a restaurant group or a SaaS company, being quoted by AI assistants is now a source of discovery. My default advice is:

  1. Leave training and grounding open unless you publish content you sell, such as paid research or courses.
  2. Do not copy AI-blocking lists from forums. Check what each line blocks, because a wrong Disallow under User-agent: * can remove you from Search entirely.
  3. If you must block training, block it for the folders that hold paid or private content, not the whole site, using Allow for public pages as in the example above.
  4. Write it down. Note in your SEO log why each AI rule exists, so the next person does not undo it by accident.

What to check this week

  • Open your live robots.txt and list every AI-related line.
  • In Search Console, check which pages are indexed and whether any were blocked by mistake.
  • Decide your policy for AI training separately from your policy for AI answers on Search.

Hyperlocal SEO AI flags robots.txt changes in its weekly technical checks, so a rule that suddenly blocks important pages shows up before rankings move.

Sources

  1. Supporting a shared AI training control for websites, Google Search Central Blog
  2. Google’s common crawlers (Google-Extended), Google Search Central
  3. AI training preference signal (Internet-Draft), IETF Datatracker
  4. Control what you share with Google Search, Google Search Central

Track all of this automatically

Hyperlocal SEO AI monitors every listing and page, every week.

Start free trial

Written by Akshay

Manages Google Business Profiles and local SEO for hospital and clinic chains across India.

Want more guides like this in your Google results?

Add us as a preferred source on Google

FAQ

FAQs

What is the shared AI training token in robots.txt?+

It is a policy token, proposed as __usage_ai-training in an IETF draft, that lets a site tell all participating AI providers whether its content may be used for training, using normal Allow and Disallow rules.

Will blocking AI training hurt my Google rankings?+

No. Google says Google-Extended does not affect inclusion in Google Search and is not used as a ranking signal.

Does blocking Google-Extended remove my site from AI Overviews?+

Google-Extended covers Gemini model training and grounding in Gemini Apps and Vertex AI. Inclusion in AI Overviews and AI Mode is handled by Search controls such as nosnippet and the Search generative AI control in Search Console.

When will Google support the shared token?+

Google’s crawler documentation says support is planned for early 2027. Until then, use Google-Extended for Google.

Working with 50+ enterprises

  • Medanta Hospitals
  • Apollo Hospitals
  • Paras Health
  • PSRI Hospital
  • Mankind Pharma
  • BSV
  • HCAH
  • QI Spine Clinic
  • Ferty9 Fertility
  • Risaa IVF
  • Dermaheal
  • Dr. Manu Gautam
  • Kornu Driving School
  • Speedy Tyres Sweden
  • Artham Derma & Orthopedic Centre
  • Renovaaura
  • Extell Systems