AI companies run separate bots for training, search indexing and user-requested fetches, and each vendor documents its own robots.txt behavior. This reference lists what each vendor states, including where user-initiated fetchers may ignore robots.txt.
Most AI vendors now split their crawling by purpose. One bot collects content that may be used for model training. Another builds a search index. A third fetches a page when a person asks a question. You can allow or block each one on its own in robots.txt, a plain text file at the root of your site that tells crawlers which paths they may fetch.
The vendor documents below are the source for every row in the table. Check them again before you change rules, because user agent versions and policies change.
The table lists the user agent token, the purpose the vendor gives, and what the vendor says about robots.txt. A blank statement means the vendor's page does not say.
| Vendor | Token | Stated purpose | robots.txt, per vendor |
|---|---|---|---|
| OpenAI | GPTBot | Crawls content that may be used to train generative AI foundation models | Disallowing it indicates content should not be used in training |
| OpenAI | OAI-SearchBot | Surfaces sites in ChatGPT search features | Opted-out sites will not be shown in ChatGPT search answers; they can still appear as navigational links |
| OpenAI | ChatGPT-User | Certain user actions in ChatGPT and Custom GPTs, such as visiting a page when a user asks | Rules may not apply, because user-initiated actions bypass them |
| Anthropic | ClaudeBot | Collects web content that could contribute to training | Blocking signals that future materials should be excluded from training datasets |
| Anthropic | Claude-User | Accesses sites when individuals ask Claude questions | Blocking prevents retrieval for user queries; Anthropic says its bots honor robots.txt |
| Anthropic | Claude-SearchBot | Navigates the web to improve search result quality | Blocking stops indexing for search optimization |
| Perplexity | PerplexityBot | Surfaces and links sites in Perplexity search results; not used for foundation model training | Perplexity recommends allowing it to appear in results |
| Perplexity | Perplexity-User | Visits a page when a user asks a question | Generally ignores robots.txt rules because a user requested the fetch |
| Google-Extended | Controls use of crawled content for training future Gemini models and grounding | A robots.txt token only; no separate HTTP user agent string | |
| Googlebot | Google's common crawler for Search | Common crawlers always obey robots.txt for automatic crawls | |
| Apple | Applebot | Powers Spotlight, Siri and Safari search features; may also help train Apple foundation models | Respects robots.txt; follows Googlebot rules if no Applebot rules exist |
| Apple | Applebot-Extended | Secondary agent that controls whether crawled data is used to train Apple foundation models | Disallow it to opt out of training; it does not crawl pages itself |
Three vendors say their user-triggered fetchers may not follow robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User. Perplexity says Perplexity-User generally ignores robots.txt rules because a user requested the fetch. Google says that because the fetch was requested by a user, its user-triggered fetchers generally ignore robots.txt rules.
Anthropic's help page says its bots honor industry standard directives in robots.txt. It lists Claude-User among its bots and does not describe a separate exception for it.
Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal, per Google. It is a control for Gemini training and grounding. Google's AI features in Search are governed by Googlebot and the snippet controls, not by Google-Extended. See our guide on Google's AI features for those controls.
Apple makes the same split. Apple says sites that disallow Applebot-Extended can still appear in search results.
Decide per purpose, not per company. A site that wants to appear in AI search answers but not in training can allow search bots and disallow training bots. For example, OpenAI says to use OAI-SearchBot to manage Search opt-outs, and GPTBot to control training.
Do not rely on robots.txt alone to stop user-initiated fetches. Vendors publish IP ranges for verification, such as OpenAI's chatgpt-user.json and Perplexity's perplexity-user.json. Anthropic warns that blocking by IP address may not guarantee an opt-out, and publishes its ranges at claude.com/crawling/bots.json. Perplexity notes that a web application firewall may need to explicitly allow its bots.
Sources were last checked on October 13, 2026. Documentation changes, so confirm any detail that matters to a decision against the source itself.
Indexive Research. (2026, September 30). Which AI crawlers honor robots.txt, by vendor. Indexive AI. https://indexive.ai/research/ai-crawlers-and-robots-txt
Straight answers about how we work. Still have a question? The free audit is the fastest way to get one answered for your site.
No. OpenAI says GPTBot relates to training. Sites opted out of OAI-SearchBot are the ones that will not be shown in ChatGPT search answers.
A free execution audit: your prioritized backlog, effort estimates, and what we'd ship first. Read-only access, no contract.