robots.txt for AI crawlers: GPTBot, ClaudeBot, PerplexityBot and Google-Extended

robots.txt for AI crawlers lets you tell each AI company's bots, one by one, whether they may read your site. The decision that matters most: you can block a company's training crawler and still allow its search crawler, but blocking the search crawler can keep you out of that assistant's answers.

Most AI companies now run more than one bot, each with its own name and job. A single "block AI" rule can cost you visibility you wanted to keep. This guide sets out what each crawler does, according to the companies' own documentation as of 11 October 2026, and gives an example robots.txt for a South African business.

In short

• OpenAI and Anthropic run separate crawlers for model training and for search, so blocking one does not block the other. Perplexity says its crawler is for search, not training.
• Blocking OAI-SearchBot means your site "will not be shown in ChatGPT search answers", in OpenAI's words. Blocking GPTBot only signals that your content should not be used for training.
• Google-Extended does not affect your inclusion or ranking in Google Search. It covers Gemini training and grounding in Gemini Apps and Vertex AI.
• User-initiated fetchers such as ChatGPT-User and Perplexity-User may not follow robots.txt at all, because a person asked for the page.
• robots.txt is a request, not a lock. The standard says its rules "are not a form of access authorization".

Which AI crawlers should you know about?

These are the robots.txt names (user-agent tokens) that matter for AI search, with what each company says about them:

Company	robots.txt name	What it does	If you block it
OpenAI	OAI-SearchBot	Surfaces websites in ChatGPT's search features	Your site is not shown in ChatGPT search answers, though it can still appear as a navigational link
OpenAI	GPTBot	Crawls content that may be used to train OpenAI's models	Signals that your content should not be used for training
OpenAI	ChatGPT-User	Visits pages when a ChatGPT user asks a question	OpenAI says robots.txt rules "may not apply"
Anthropic	ClaudeBot	Collects web content that could contribute to model training	Signals that your future content should be excluded from training datasets
Anthropic	Claude-SearchBot	Indexes content to improve Claude's search results	May reduce your visibility in Claude's search results
Anthropic	Claude-User	Fetches pages when a Claude user asks a question	Claude cannot retrieve your pages for user questions
Perplexity	PerplexityBot	Surfaces and links websites in Perplexity's search results; not used to train foundation models	Perplexity recommends allowing it "to ensure your site appears in search results"
Perplexity	Perplexity-User	Visits pages when a Perplexity user asks a question	Perplexity says it "generally ignores robots.txt rules"
Google	Google-Extended	Controls use of content for Gemini training and grounding	No effect on Google Search inclusion or ranking

What does blocking each AI crawler actually do?

OpenAI. OpenAI says "each setting is independent of the others". Its own example is a site that allows OAI-SearchBot to appear in search results while disallowing GPTBot. If you allow both, OpenAI may use one crawl for both purposes. ChatGPT-User is not used to decide whether content appears in search; OAI-SearchBot is the setting for that.

Anthropic. Anthropic runs three bots and says they honour robots.txt. Blocking ClaudeBot keeps your future content out of training. Blocking Claude-SearchBot or Claude-User "may reduce" your visibility in Claude's answers. Anthropic also supports the non-standard Crawl-delay rule if you want it to crawl more slowly, and warns that blocking its IP addresses instead of using robots.txt "may not work correctly".

Perplexity. PerplexityBot is Perplexity's search crawler, and Perplexity states it is not used to crawl content for AI foundation models. Perplexity-User fetches pages for a user's question and generally ignores robots.txt. If your firewall blocks bots, Perplexity advises allowing its crawlers and its published IP ranges.

Google. Google-Extended has no crawler of its own. Google crawls with its usual user agents and reads the Google-Extended rule "in a control capacity". It governs whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google states that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal".

Does blocking Google-Extended keep you out of AI Overviews?

No. AI Overviews are part of Google Search, and Google-Extended does not affect Google Search. To limit what Google shows from your pages, including in its AI features, Google points to nosnippet, data-nosnippet, max-snippet and noindex. Those controls change your ordinary search results too, and Google documents no robots.txt rule that removes you from AI Overviews alone. For more on how Overviews choose sources, read Google AI Overviews explained.

An example robots.txt for AI crawlers

For example, a South African insurer that wants to appear in AI search answers but keep its content out of model training could publish the file below. It is an illustration of one policy, not a recommendation for every business.

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /account/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

Three details make this work:

1. A crawler obeys the group that names it. Under the robots.txt standard (RFC 9309), a crawler looks for a group matching its name and falls back to the User-agent: * group only if none exists. So the search crawlers above do not inherit the * rules, which is why /account/ is repeated in their group.
2. Several names can share one group. Listing three user agents above one set of rules is valid and keeps the file short.
3. The most specific rule wins. When an Allow and a Disallow both match a URL, the standard says the longer, more specific rule applies.

Googlebot falls under the * group here, so ordinary Google Search, including AI Overviews, is unaffected.

What can't robots.txt do?

• Stop a determined bot. RFC 9309 says the rules "are not a form of access authorization". Pages that must stay private need a login.
• Control every user-initiated fetch. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Anthropic, by contrast, says Claude-User lets site owners control which sites these user-initiated requests can access.
• Act instantly. OpenAI says about 24 hours for search results; Perplexity says up to 24 hours. The standard says crawlers should not use a cached copy for more than 24 hours.
• Override your firewall. If your CDN or firewall blocks bots, an Allow rule in robots.txt changes nothing. Check both.
• Make crawlers read your content. A crawler that is allowed in still needs the words in your HTML. If your site builds its text with JavaScript, see is your website readable by ChatGPT?

How should you decide what to allow?

1. List the assistants your buyers use. If they use ChatGPT, Claude or Perplexity to research purchases, their search crawlers are worth allowing.
2. Decide on training separately. Blocking GPTBot, ClaudeBot or Google-Extended is a policy choice about training and grounding, not a search choice. Note that Google-Extended also covers grounding in Gemini Apps.
3. Fetch your live file. Open https://yoursite/robots.txt in a browser and check for a blanket Disallow: / under * or under a search crawler's name.
4. Ask your hosting or CDN provider whether bot protection is blocking AI crawlers before they ever see robots.txt.
5. Measure after you change it. Ask the same buyer questions before and after, on each assistant, and compare whether you are named and cited.

robots.txt only controls access. Files such as llms.txt are a separate, voluntary convention, and neither decides whether an assistant recommends you.