How AI answers pick sources · Official documentation

4 of 7 AI companies say robots.txt may not apply to their user-triggered fetchers

DATAPREPREP · data as of 2026-09-14 · 4 min read한국어로 읽기

Not according to OpenAI's documentation: GPTBot controls training, and ChatGPT search answers are controlled by OAI-SearchBot. Comparing the crawler documentation of 7 AI companies, 5 keep an AI training token separate from their search crawler, and 4 of the 5 with a user-triggered fetcher say robots.txt may not apply to it.

Blocking or allowing AI crawlers usually means listing a few user-agent tokens in robots.txt. Tokens that sit side by side in that file do different jobs, according to the companies that run them.

We opened the crawler documentation published by OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, and Meta and compared, user agent by user agent, what each one does and how far robots.txt applies to it.

Finding 01

5 companies control AI training with a token separate from their search crawler

Each company separates its user agents by purpose in different ways

Separate user agent documented
Another method documented instead
Not in the documentation

Each cell is based on the company's own crawler documentation. The user-agent names themselves are in the table below.

Company	Search / answers	AI training	User-triggered
OpenAI	OAI-SearchBot	GPTBot	ChatGPT-User
Anthropic	Claude-SearchBot	ClaudeBot	Claude-User
Perplexity	PerplexityBot	No AI training token; says its search crawler is not used for AI model training	Perplexity-User
Google	Googlebot	Google-Extended	Google-Agent
Microsoft	Bingbot	No user-agent token; says it uses the NOARCHIVE meta tag instead	Not stated
Apple	Applebot	Applebot-Extended	Not stated
Meta	Meta-WebIndexer	Meta-ExternalAgent	Meta-ExternalFetcher

Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI’s generative AI foundation models.

OpenAI, "Overview of OpenAI Crawlers"

The 2 companies without an AI training token explain why. Perplexity says its search crawler is not used to crawl content for AI models, and Microsoft says it separates that use with the NOARCHIVE meta tag rather than a user-agent token.

Finding 02

At 4 of 5 companies, user-triggered fetchers may not follow robots.txt

Company	User-triggered fetcher	Original sentence
OpenAI	ChatGPT-User	Because these actions are initiated by a user, robots.txt rules may not apply.
Anthropic	Claude-User	Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.
Perplexity	Perplexity-User	Since a user requested the fetch, this fetcher generally ignores robots.txt rules.
Google	Google-Agent	Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules.
Meta	Meta-ExternalFetcher	Accordingly, this crawler may bypass robots.txt rules.

These user agents fetch a page when a user pastes a URL or asks a question in the chat interface.

Since a user requested the fetch, this fetcher generally ignores robots.txt rules.

Perplexity, "Perplexity Crawlers"

Crawling for a search index and fetching a page because a user asked are two different paths. Index crawlers follow robots.txt; for user-triggered fetches, most of these documents say the file may not apply.

Anthropic writes that disabling Claude-User stops it from retrieving your content in response to a user query. Fetchers in the same category are described differently from one company to the next.

Finding 03

Google-Extended and Applebot-Extended are control tokens, not crawlers

Google-Extended doesn't have a separate HTTP request user agent string.

Google Search Central, "Google's common crawlers"

Applebot-Extended does not crawl webpages.

Apple, "About Applebot"

Search your server logs for these tokens and you will not find them. The crawling is done by Googlebot and Applebot; the control tokens only decide whether what they crawled may be used for AI training.

Finding 04

Bing says Copilot answers are controlled by a meta tag, not a user-agent token

NOARCHIVE prevents content from being used in Copilot responses and grounding results.

Microsoft Bing, "Bing Webmaster Guidelines"

Bing's guidelines say Bing search and Copilot run on the same crawling, indexing and ranking foundation. That is why the document names no user agent that controls Copilot on its own.

So what goes in robots.txt?

Define search and answer crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and others) and AI training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and others) as separate user-agent groups.
Plan for the possibility that user-triggered fetchers are not stopped by robots.txt alone.
For Bing Copilot answers, use the NOARCHIVE meta tag on the pages concerned.

Methodology

OpenAI, "Overview of OpenAI Crawlers" · checked 2026-09-14 · https://developers.openai.com/api/docs/bots
Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?" (updated 2026-04-07) · checked 2026-09-14 · https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
Perplexity, "Perplexity Crawlers" · checked 2026-09-14 · https://docs.perplexity.ai/docs/resources/perplexity-crawlers
Google Search Central, "Google's common crawlers" (updated 2026-07-14) · checked 2026-09-14 · https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
Google Search Central, "Google User-Triggered Fetchers" (updated 2026-08-19) · checked 2026-09-14 · https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers
Microsoft Bing, "Bing Webmaster Guidelines" · checked 2026-09-14 · https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a
Apple, "About Applebot" (updated 2026-09-04) · checked 2026-09-14 · https://support.apple.com/en-us/119829
Meta, "Meta Web Crawlers" · checked 2026-09-14 · https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers
Quotes are reproduced in the original English. Only the table labels and cell judgments are ours.
This covers what each company's documentation says. Whether the crawlers actually behave that way has to be checked separately in server logs.
Google has several user-triggered fetcher names. The table lists Google-Agent as the representative, and the quoted sentence applies to that whole document.