Google Search · Official documentation
Google's controls over AI answers sit in 4 different places, and robots.txt is only one of them
robots.txt is one of 4 Google-side controls. According to Google's documentation, Google-Extended does not affect inclusion in Google Search, and a separate Search Console control for opting out of AI Overviews and AI Mode has been rolled out to all websites as of August 31, 2026.
Many site owners add AI crawler user agents to robots.txt to stay out of AI answers; others allow every crawler to get into them. Both moves often assume one setting controls everything.
We checked Google's own documents for what each control actually changes.
Finding 01
Search results, AI Overviews and Gemini training are governed by different controls
- Affected
- Not affected
- Not stated in the document
Every cell rests on a sentence in Google's document for that control. Cells the documents don't address are left empty.
| Control | Where | What it changes | What stays the same |
|---|---|---|---|
| Googlebot rules in robots.txt | robots.txt at the site root | Crawling for all of Google Search; AI Overviews and AI Mode are part of it | Blocking it removes the site from regular search results too, not only from AI answers |
| Google-Extended | A separate token in robots.txt | Gemini model training, and grounding in the Gemini app and Vertex AI | Inclusion and ranking in Google Search stay as they are |
| nosnippet · max-snippet · noindex | robots meta tag on the page | How much of the page is shown as a snippet; noindex removes the page from Search | Works page by page, not for the whole site at once |
| Search generative AI control | Search Console settings | Whether the site is included in AI Overviews, AI Mode and generative features in Discover | Regular search visibility, ranking and model training are unaffected |
Finding 02
Blocking Google-Extended does not take a site out of Google Search
Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.
The token governs Gemini model training and grounding in the Gemini app and Vertex AI. AI Overviews and AI Mode are features inside Google Search, so they follow the Googlebot rules instead.
Finding 03
The opt-out for AI Overviews and AI Mode alone lives in Search Console
As of August 31, 2026, we've rolled out this control to all websites worldwide. This control only affects whether your content can appear in certain Search generative AI features; this control isn't used as a ranking or inclusion signal affecting other parts of Search.
The default is included. According to the help page, switching to excluded usually takes effect within a few days and does not touch regular search visibility or model training.
So which Google control matches which goal?
- To leave only Google's AI answers: set the Search Console generative AI control to excluded.
- To leave only Gemini training and grounding: disallow Google-Extended in robots.txt.
- To keep part of a page out of snippets: apply nosnippet or max-snippet to that page.
- To be eligible for AI answers: keep Googlebot allowed and leave the Search Console control on included.
Methodology
- Google Search Central, "AI features and your website" (updated 2025-12-10) · checked 2026-09-13 · https://developers.google.com/search/docs/appearance/ai-features
- Google Search Central, "Google's common crawlers (Google-Extended)" (updated 2026-07-14) · checked 2026-09-13 · https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- Search Console Help, "Search generative AI control" · checked 2026-09-13 · https://support.google.com/webmasters/answer/16908024
- Each control's effect is stated only as far as Google's documents state it. The Search Console help page says changes usually apply within a few days.
- This article covers Google's controls. Other AI companies document their own crawler user agents and scopes separately.