agenticweb.wiki

AI Crawler User-Agent Tokens in robots.txt

Permit This record covers the most widely deployed permit mechanism on the web today: a family of vendor-specific user-agent tokens that AI companies define for use inside a site's existing robots.txt file, built on the IETF's RFC 9309 Robots Exclusion Protocol syntax. It is not a single spec but a convention each vendor documents and maintains independently, unified only by shared syntax (User-agent groups with allow/disallow rules) [OpenAI, Anthropic, Google, Perplexity docs below].

Overview

Four major vendors were checked directly for this record. OpenAI documents GPTBot (training data collection), OAI-SearchBot (search indexing), and OAI-AdsBot (ad-related validation), each with its own user-agent string and published IP range for verification [1]. Anthropic documents ClaudeBot (training-data collection), Claude-User (fetches triggered by a live user question to Claude), and Claude-SearchBot (search-quality crawling), explaining separately what disabling each one via robots.txt does to a site's treatment [2]. Google documents the long-standing Googlebot token alongside the AI-specific Google-Extended token, and is explicit that Google-Extended controls only training data for Gemini models and Vertex AI grounding, with 'no effect on Google Search or other products' — confirming, directly from Google's own text, the prior finding that Google-Extended does not control Search or AI Overviews [3]. Perplexity documents PerplexityBot (crawling for its models) and Perplexity-User, which it states explicitly 'is not used for web crawling or to collect content for training AI foundation models' but instead fetches pages live in response to a user's question [4].

The pattern across all four vendors is the same: a 'bulk crawling / training' token and a separate 'live, user-triggered fetch' token, plus sometimes a third 'search index' token, each independently blockable. This granularity is exactly the gap the IETF AIPREF drafts and Cloudflare's Content Signals Policy are trying to standardize into a single, vendor-neutral vocabulary — today, a publisher who wants to block AI training but keep search visibility must maintain a separate robots.txt group per vendor, using each vendor's own token names, rather than one general-purpose signal.

As of this verification (2026-09-06), all four vendor documentation pages were live and internally consistent with each other's pattern. No unified registry of every AI crawler token across every vendor (beyond each vendor's own page) was checked in this pass; adoption is recorded as observed, since the tokens themselves are demonstrably deployed by name-brand vendors and honoured by a large fraction of the web's robots.txt files, but no systematic probe run counted that fraction for this record. What is unresolved: whether the IETF AIPREF or Cloudflare Content Signals conventions will eventually consolidate these per-vendor tokens into one general vocabulary, or whether new AI companies will simply keep adding their own named tokens to this already-fragmented list.

Discovery and probe

User-agent group directives inside robots.txt, one group per named crawler token · /robots.txt

GET {origin}/robots.txt
parse:     Look for User-agent groups matching known tokens: GPTBot, OAI-SearchBot, OAI-AdsBot (OpenAI); ClaudeBot, Claude-User, Claude-SearchBot (Anthropic); Googlebot, Google-Extended (Google); PerplexityBot, Perplexity-User (Perplexity); then read the allow/disallow rules in that group
on absent: No group for a given token means that crawler applies its default (documented per-vendor, typically full access) rather than an explicit restriction

Live check run by the MCP probe: GET /robots.txt. A robots.txt group naming a known AI crawler token.

{
 "path": "/robots.txt",
 "expect": {
  "text_match": "(?im)^user-agent:\\s*(GPTBot|OAI-SearchBot|ClaudeBot|Claude-User|Claude-SearchBot|Google-Extended|PerplexityBot|CCBot|Applebot-Extended|meta-externalagent|Bytespider|Amazonbot)\\b"
 },
 "note": "A robots.txt group naming a known AI crawler token."
}

History

Instances

Every instance with its source is listed on the instances page.

See also

References

  1. OpenAI's own bot documentation (developers.openai.com/api/docs/bots) names OAI-SearchBot, OAI-AdsBot, and GPTBot with distinct user-agent strings and published IP ranges, and recommends site owners control each via robots.txt. — https://developers.openai.com/api/docs/bots (2026) VERIFIED
  2. Anthropic's support article names three distinct robots — ClaudeBot (training), Claude-User (user-triggered access), and Claude-SearchBot (search quality) — and documents what disabling each one via robots.txt does. — https://support.claude.com/en/articles/8896518 (2026) VERIFIED
  3. Google's documentation states that Google-Extended is a standalone robots.txt product token controlling only Gemini model training/grounding and Vertex AI, and explicitly states it has no effect on Google Search or other Google products, confirming it is distinct from the Googlebot token. — https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (2026) VERIFIED
  4. Perplexity's crawler documentation names PerplexityBot (crawling/training) and Perplexity-User (live user-query-triggered fetches, explicitly not used for crawling or training) as separate tokens with separate published IP ranges. — https://docs.perplexity.ai/guides/bots (2026) VERIFIED

JSON · Markdown