agenticweb.wiki

AI crawler tokens

By Belgravia Labs Standards Observatory · Last verified · 3 primary sources

AI crawler tokens are the User-agent names AI companies document for use in robots.txt under RFC 9309, split by purpose rather than treated as one identity. In the agentic web (Step 3, Permit), also known as crawler user-agent tokens, robots.txt AI bots, it is implemented or referenced by Content Signals Policy.

Permit AI crawler tokens are the User-agent names AI companies document for use in robots.txt under RFC 9309, split by purpose rather than treated as one identity. OpenAI documents OAI-SearchBot (search inclusion in ChatGPT), GPTBot (training data crawling), OAI-AdsBot (ad-landing-page checks) and ChatGPT-User (live fetches triggered by a user's own request, which is not governed by the search or training tokens). Anthropic and Perplexity publish an equivalent three-way split for their own products. Disallowing one token does not affect the others: a site can, for example, permit search indexing while refusing to have its content used for model training. ChatGPT-User- and Perplexity-User-class tokens are user-triggered fetches, not automated crawling, so robots.txt rules may not apply to them the way they do to a crawler. Used by: cloudflare-content-signals-policy.

Overview

2026 AI Crawler & User-Triggered Fetcher Reference

Token Operator Role (Index / Live Fetch / Training) Honors robots.txt Verification / Notes
OAI-SearchBot OpenAI Index (ChatGPT Search) Yes Published IP ranges (openai.com/searchbot.json); separate from training
ChatGPT-User OpenAI Live Fetch (user-triggered) Yes (user-agent rule) Published IP ranges (openai.com/chatgpt-user.json); acts on explicit user prompt
GPTBot OpenAI Training Yes Published IP ranges (openai.com/gptbot.json); disallowing blocks foundation model training
OAI-AdsBot OpenAI Index (ad landing-page quality) Yes Evaluates destination URLs for ChatGPT ad placements
ClaudeBot Anthropic Training Yes Crawls public web data for Claude model training
Claude-SearchBot Anthropic Index (Claude web search) Yes Indexes pages for citations in Claude search results
Claude-User Anthropic Live Fetch (user-triggered) Yes (user-agent rule) Fetches URLs requested by a Claude user during a session
PerplexityBot Perplexity Index (Perplexity answer engine) Yes Published IP list (perplexity.ai/perplexitybot.json); search indexing
Perplexity-User Perplexity Live Fetch (user-triggered) Disputed / Partial User-initiated page retrieval; RFC 9309 applicability to user agents is contested
Googlebot Google Index (Google Search & AI Overviews) Yes Reverse DNS (*.googlebot.com) & IP ranges; controls Search and grounding
Google-Extended Google Training (Gemini models & Vertex AI) Yes Controls Gemini training/fine-tuning; does not remove pages from Google Search or AI Overviews
Google-Agent Google Live Fetch / Action (Project Mariner & Gemini agents) Browser / Session Identifies browser-mediated agentic sessions (often paired with Web Bot Auth signatures)
bingbot Microsoft Index (Bing Search & Copilot grounding) Yes Reverse DNS (*.search.msn.com) & published IP list; powers Copilot citations
Applebot-Extended Apple Training (Apple Intelligence) Yes Separate from Applebot search index; opt-out of Apple foundation model training
Amazonbot Amazon Index & Training (Alexa / Rufus / Q) Yes Reverse DNS (*.crawl.amazonbot.amazon); supports standard robots.txt rules
CCBot Common Crawl Training (open web corpus) Yes Public dataset widely ingested by third-party LLM pre-training pipelines
Meta-ExternalAgent Meta Training & Index (Meta AI) Yes Matches case-insensitively (meta-externalagent); used for Meta AI training and retrieval

Copy-Paste robots.txt: Allow AI Search & User Agents, Block Model Training

# 1. Allow AI search indexers & live user-triggered fetchers
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Googlebot
User-agent: Google-Agent
User-agent: bingbot
Allow: /

# 2. Block foundation-model training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: CCBot
User-agent: Meta-ExternalAgent
Disallow: /

# 3. Default policy & discovery pointers
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Sitemap: https://example.com/sitemap.xml
Agentmap: https://example.com/.well-known/ard.json

Instances

Direct protocol probe: Not applicable (general term). Verified implementations across related standards:

Every instance with its source is listed on the instances page.

See also

References

  1. https://developers.openai.com/api/docs/bots (2026-09-05) VERIFIED
  2. https://support.claude.com/en/articles/8896518 (2026-04-07) VERIFIED
  3. https://docs.perplexity.ai/guides/bots (2026-09-05) REPORTED

JSON · Markdown · OKF v0.2

Frequently asked questions

What is AI crawler tokens?
AI crawler tokens are the User-agent names AI companies document for use in robots.txt under RFC 9309, split by purpose rather than treated as one identity. In the agentic web (Step 3, Permit), also known as crawler user-agent tokens, robots.txt AI bots, it is implemented or referenced by Content Signals Policy.
Which agentic web standards use AI crawler tokens?
AI crawler tokens is used or referenced by Content Signals Policy. See https://agenticweb.wiki/terms/ai-crawler-tokens/ for sourced definitions and verified implementations.
How is AI crawler tokens verified on agenticweb.wiki?
The AI crawler tokens record cites 3 primary specification or implementation sources and was last verified on 2026-10-30.