agenticweb.wiki

AI crawler tokens in robots.txt

Last checked · 22 primary sources

This page covers the vendor-published user-agent and product tokens that AI companies tell site owners to name in robots.txt. It is not about the Robots Exclusion Protocol itself (RFC 9309), Cloudflare's Content Signals Policy, the IETF AIPREF vocabulary, ai.txt, agents.txt or llms.txt. Those are separate files or vocabularies; this page is the list of vendor names and what each vendor says the name controls.

Overview

AI vendors publish named tokens so that a site owner can write a robots.txt group for one vendor's crawler instead of for every robot. Several vendors now split their names by purpose, with one token for collecting training data, another for building a search index and a third for fetches a user asks for, so a site can refuse training while staying visible in AI search. Some names, such as Google-Extended and Applebot-Extended, are not crawlers at all but control tokens that govern how data fetched by the vendor's ordinary crawler may be used. RFC 9309 is clear that robots.txt rules ask rather than enforce, and several vendors say their user-initiated fetchers may not follow them.

RFC 9309: "These rules are not a form of access authorization." [1]

Tokens by vendor

Purpose classes: Training = the vendor says the token controls use for model training; Search/index = the vendor says it is for search or answer indexing and not training; User fetch = fetches made because a user asked. Only tokens whose vendor page was fetched on 2026-10-29 are listed.

Vendor Token Class Vendor's own words Page date
OpenAI GPTBot Training "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." [2] not given
OpenAI OAI-SearchBot Search/index "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." [2] not given
OpenAI ChatGPT-User User fetch "ChatGPT-User is not used for crawling the web in an automatic fashion." [2] not given
OpenAI OAI-AdsBot Other (ads review), not training "OAI-AdsBot only visits pages submitted as ads, and the data collected by OAI-AdsBot is not used to train generative AI foundation models." [2] not given
Anthropic ClaudeBot Training "When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets." [3] "April 7, 2026"
Anthropic Claude-SearchBot Search/index "Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results." [3] "April 7, 2026"
Anthropic Claude-User User fetch "Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search." [3] "April 7, 2026"
Google Google-Extended Training (and Gemini grounding); control token, no own crawler "Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI." [4] "Last updated 2026-10-15 UTC."
Google Google-Agent User fetch "Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request." [5] "Last updated 2026-08-19 UTC."
Perplexity PerplexityBot Search/index "PerplexityBot is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models." [6] not given
Perplexity Perplexity-User User fetch "Perplexity-User controls which sites these user requests can access. It is not used for web crawling or to collect content for training AI foundation models." [6] not given
Apple Applebot-Extended Training; control token, no own crawler "Web publishers can opt-out from having their content used to train generative foundation models by disallowing Applebot-Extended in the robots.txt file." [7] "Published Date: September 04, 2026"
Meta Meta-ExternalAgent Training (and product indexing) "The Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly." [8] not given (Wayback capture 2026-06-16)
Meta Meta-ExternalFetcher User fetch "The Meta-ExternalFetcher crawler may bypass robots.txt because it performs fetches that were requested by the user." [8] not given (Wayback capture 2026-06-16)
Common Crawl CCBot Open crawl archive (used downstream for training) "To prevent Common Crawl from crawling your website, include the following in your robots.txt: User-agent: CCBot Disallow: /" [9] not given
Amazon Amazonbot Training (among other uses) "Amazonbot is used to improve our products and services. This helps us provide more accurate information to customers and may be used to train Amazon AI models." [10] "Last updated: Oct 26, 2026"
Amazon Amzn-SearchBot Search/index "Amzn-SearchBot does not crawl content for generative AI model training." [10] "Last updated: Oct 26, 2026"
Amazon Amzn-User User fetch "Because actions taken by Amzn-User can be initiated by a user, it may not follow all robots.txt directives." [10] "Last updated: Oct 26, 2026"
Mistral AI MistralAI-Training Training "MistralAI-Training crawls web content to help build datasets for training Mistral generative AI models. Webmasters can disallow this user agent in their robots.txt file." [11] not given
Mistral AI MistralAI-Index Search/index "Content crawled by MistralAI-Index is not used for generative AI training of any kind." [11] not given
Mistral AI MistralAI-User User fetch "It is not used for crawling the web in any automatic fashion, nor to crawl content for generative AI training." [11] not given
DuckDuckGo DuckAssistBot Search/index (AI answers) "DuckAssistBot is a web crawler for DuckDuckGo Search that crawls pages in real-time for our AI-assisted answers, which prominently cite their sources. This data is not used in any way to train AI models." [12] not given
Ai2 AI2Bot Training "This web content is used to train open language models." [13] not given

Vendors that say user-initiated fetchers may ignore robots.txt

Who uses it

Examples from live robots.txt files fetched on 2026-10-29 (EET), with lines copied by script; each group is shown up to 10 lines.

Live count. The sample was a fixed list of 36 well-known news, reference and platform hosts that served robots.txt with HTTP 200 and no redirect on 2026-10-29. Forty hosts were tried; www.cnn.com redirected and was replaced by edition.cnn.com, and four were dropped: www.usatoday.com (302 redirect), www.politico.com (403), stackoverflow.com (418) and www.yelp.com (403). Of these, 32 name at least one of the 13 documented tokens below in a User-agent line, and 29 put at least one of them in a group with Disallow: /. "Disallow: /" here means the group contains Disallow: / and no Allow: /; narrower Allow exceptions may still apply. This is a convenience sample, not a statistical one.

Token Sites naming it Sites with Disallow: /
GPTBot 23 / 36 19 / 36
OAI-SearchBot 18 / 36 12 / 36
ChatGPT-User 18 / 36 15 / 36
Google-Extended 19 / 36 16 / 36
ClaudeBot 26 / 36 25 / 36
Claude-User 20 / 36 19 / 36
Claude-SearchBot 20 / 36 18 / 36
PerplexityBot 22 / 36 21 / 36
Perplexity-User 14 / 36 14 / 36
Applebot-Extended 23 / 36 23 / 36
Meta-ExternalAgent 19 / 36 19 / 36
CCBot 24 / 36 24 / 36
Amazonbot 20 / 36 16 / 36

Who reads it

Vendor statements about honouring robots.txt. These are what the vendors say, not measured compliance.

Recent coverage

See also

References

  1. RFC 9309, Robots Exclusion Protocol (IETF, September 2022), checked 2026-10-29: "These rules are not a form of access authorization." — https://www.rfc-editor.org/rfc/rfc9309.html
  2. OpenAI, Overview of OpenAI Crawlers, checked 2026-10-29: "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." "ChatGPT-User is not used for crawling the web in an automatic fashion." "OAI-AdsBot only visits pages submitted as ads, and the data collected by OAI-AdsBot is not used to train generative AI foundation models." "Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI's generative AI foundation models." "Because these actions are initiated by a user, robots.txt rules may not apply." "OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI." — https://developers.openai.com/api/docs/bots
  3. Anthropic (Claude Help Center), Does Anthropic crawl data from the web, and how can site owners block the crawler?, April 7, 2026, checked 2026-10-29: "April 7, 2026" "When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets." "Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results." "Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search." "Anthropic's Bots respect "do not crawl" signals by honoring industry standard directives in robots.txt." — https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
  4. Google Search Central, Google's common crawlers, Last updated 2026-10-15 UTC, checked 2026-10-29: "Last updated 2026-10-15 UTC." "Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI." "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." "Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." "They always obey robots.txt rules when crawling automatically." — https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
  5. Google Search Central, Google's user-triggered fetchers, Last updated 2026-08-19 UTC, checked 2026-10-29: "Last updated 2026-08-19 UTC." "Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request." "Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules." — https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers
  6. Perplexity, Perplexity Crawlers, checked 2026-10-29: "PerplexityBot is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models." "Perplexity-User controls which sites these user requests can access. It is not used for web crawling or to collect content for training AI foundation models." "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." — https://docs.perplexity.ai/docs/resources/perplexity-crawlers
  7. Apple Support, About Applebot, Published Date: September 04, 2026, checked 2026-10-29: "Published Date: September 04, 2026" "Web publishers can opt-out from having their content used to train generative foundation models by disallowing Applebot-Extended in the robots.txt file." "Applebot-Extended does not crawl webpages. Webpages that disallow Applebot-Extended can still be included in search results." "Applebot respects standard robots.txt directives in general search crawls that are targeted at Applebot." — https://support.apple.com/en-us/119829
  8. Meta for Developers, Meta Web Crawlers (Wayback Machine capture 2026-06-16 00:58:30 UTC; live page blocks fetches), checked 2026-10-29: "The Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly." "The Meta-ExternalFetcher crawler may bypass robots.txt because it performs fetches that were requested by the user." "In order to block these crawlers, add a disallow for the relevant crawler to robots.txt." — https://web.archive.org/web/20260616005830id_/https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
  9. Common Crawl, CCBot, checked 2026-10-29: "To prevent Common Crawl from crawling your website, include the following in your robots.txt: User-agent: CCBot Disallow: /" — https://commoncrawl.org/ccbot
  10. Amazon Developer, Amazonbot, Last updated: Oct 26, 2026, checked 2026-10-29: "Last updated: Oct 26, 2026" "Amazonbot is used to improve our products and services. This helps us provide more accurate information to customers and may be used to train Amazon AI models." "Amzn-SearchBot does not crawl content for generative AI model training." "Because actions taken by Amzn-User can be initiated by a user, it may not follow all robots.txt directives." "Automated crawling from these listed user agents respects the Robots Exclusion Protocol, honoring the user-agent and the allow/disallow directives." — https://developer.amazon.com/docs/amazonbot/amazonbot.html
  11. Mistral AI docs, Robots, checked 2026-10-29: "MistralAI-Training crawls web content to help build datasets for training Mistral generative AI models. Webmasters can disallow this user agent in their robots.txt file." "Content crawled by MistralAI-Index is not used for generative AI training of any kind." "It is not used for crawling the web in any automatic fashion, nor to crawl content for generative AI training." — https://docs.mistral.ai/robots
  12. DuckDuckGo Help Pages, DuckAssistBot, checked 2026-10-29: "DuckAssistBot is a web crawler for DuckDuckGo Search that crawls pages in real-time for our AI-assisted answers, which prominently cite their sources. This data is not used in any way to train AI models." — https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot
  13. Ai2, Crawler, checked 2026-10-29: "This web content is used to train open language models." — https://allenai.org/crawler
  14. Cohere docs, Cohere Web Crawlers, checked 2026-10-29: "It is Cohere's policy to require that crawlers be designed to respect robots.txt." — https://docs.cohere.com/docs/cohere-web-crawlers
  15. www.nytimes.com robots.txt (fetched 2026-10-29), checked 2026-10-29: "User-agent: GPTBot Disallow: /" "User-agent: OAI-SearchBot Disallow: /" — https://www.nytimes.com/robots.txt
  16. www.bbc.co.uk robots.txt (fetched 2026-10-29), checked 2026-10-29: "User-agent: Google-Extended Disallow: /" "User-agent: ChatGPT-User Disallow: /" — https://www.bbc.co.uk/robots.txt
  17. www.theatlantic.com robots.txt (fetched 2026-10-29), checked 2026-10-29: "User-agent: ClaudeBot Disallow: /" "User-agent: GPTBot Disallow: /4624/TheAtlanticOnline/*" — https://www.theatlantic.com/robots.txt
  18. www.vox.com robots.txt (fetched 2026-10-29), checked 2026-10-29: "User-agent: GPTBot Allow: /" "User-agent: ClaudeBot Disallow: / Allow: /sp/ Allow: /strategist" — https://www.vox.com/robots.txt
  19. github.com robots.txt (fetched 2026-10-29), checked 2026-10-29: "User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: PerplexityBot Crawl-delay: 1" — https://github.com/robots.txt
  20. Search Engine Journal (Matt G. Southern), Google Explains The Limit Of Robots.txt For AI Content Use, October 22, 2026, checked 2026-10-29: "Matt G. Southern" "October 22, 2026" "Google-Extended is a robots.txt token with no user agent of its own." — https://www.searchenginejournal.com/google-illyes-robots-txt-crawling-ai-training/593957/
  21. SearchDock (Umang Thakar), State of AI Crawler Access 2026, Scanned October 20, 2026, checked 2026-10-29: "Scanned October 20, 2026" "27.2% of the 555 reachable websites block at least one AI crawler." "27.2% of the top websites block at least one AI crawler, in robots.txt or at the server." — https://searchdock.io/research/ai-crawler-access-2026/
  22. Kavel, Which big websites block AI crawlers?, October 2026 check, finished 22 October 2026, checked 2026-10-29: "October 2026 check, finished 22 October 2026." "We checked the robots.txt rules of 945 well-known websites for 14 AI crawlers: news, retail, SaaS, finance, government and more." "robots.txt is a request, not a lock." "This page was built and written with AI." — https://getkavel.com/ai-crawlers/

JSON · Markdown · OKF v0.2