---
type: Glossary Term
title: "AI crawler tokens"
description: "AI crawler tokens are the User-agent names AI companies document for use in robots.txt under RFC 9309, split by purpose rather than treated as one identity."
resource: "https://agenticweb.wiki/terms/ai-crawler-tokens/"
tags: ["glossary", "permit"]
status: stable
generated: { by: agenticweb-builder/1.0, at: 2026-10-30T00:00:00Z }
verified: { by: process:agenticweb-engine, at: 2026-10-30T00:00:00Z }
stale_after: 2026-11-29T00:00:00Z
sources:
  - id: ref-1
    resource: "https://developers.openai.com/api/docs/bots"
    last_modified: 2026-09-05T00:00:00Z
  - id: ref-2
    resource: "https://support.claude.com/en/articles/8896518"
    last_modified: 2026-04-07T00:00:00Z
  - id: ref-3
    resource: "https://docs.perplexity.ai/guides/bots"
    last_modified: 2026-09-05T00:00:00Z
---

# AI crawler tokens

**AI crawler tokens** are the `User-agent` names AI companies document for use in `robots.txt` under RFC 9309, split by purpose rather than treated as one identity. OpenAI documents OAI-SearchBot (search inclusion in ChatGPT), GPTBot (training data crawling), OAI-AdsBot (ad-landing-page checks) and ChatGPT-User (live fetches triggered by a user's own request, which is not governed by the search or training tokens). Anthropic and Perplexity publish an equivalent three-way split for their own products. Disallowing one token does not affect the others: a site can, for example, permit search indexing while refusing to have its content used for model training. `ChatGPT-User`- and `Perplexity-User`-class tokens are user-triggered fetches, not automated crawling, so robots.txt rules may not apply to them the way they do to a crawler. Used by: cloudflare-content-signals-policy.

### 2026 AI Crawler & User-Triggered Fetcher Reference

| Token | Operator | Role (Index / Live Fetch / Training) | Honors robots.txt | Verification / Notes |
| --- | --- | --- | --- | --- |
| `OAI-SearchBot` | OpenAI | Index (ChatGPT Search) | Yes | Published IP ranges (`openai.com/searchbot.json`); separate from training |
| `ChatGPT-User` | OpenAI | Live Fetch (user-triggered) | Yes (user-agent rule) | Published IP ranges (`openai.com/chatgpt-user.json`); acts on explicit user prompt |
| `GPTBot` | OpenAI | Training | Yes | Published IP ranges (`openai.com/gptbot.json`); disallowing blocks foundation model training |
| `OAI-AdsBot` | OpenAI | Index (ad landing-page quality) | Yes | Evaluates destination URLs for ChatGPT ad placements |
| `ClaudeBot` | Anthropic | Training | Yes | Crawls public web data for Claude model training |
| `Claude-SearchBot` | Anthropic | Index (Claude web search) | Yes | Indexes pages for citations in Claude search results |
| `Claude-User` | Anthropic | Live Fetch (user-triggered) | Yes (user-agent rule) | Fetches URLs requested by a Claude user during a session |
| `PerplexityBot` | Perplexity | Index (Perplexity answer engine) | Yes | Published IP list (`perplexity.ai/perplexitybot.json`); search indexing |
| `Perplexity-User` | Perplexity | Live Fetch (user-triggered) | Disputed / Partial | User-initiated page retrieval; RFC 9309 applicability to user agents is contested |
| `Googlebot` | Google | Index (Google Search & AI Overviews) | Yes | Reverse DNS (`*.googlebot.com`) & IP ranges; controls Search and grounding |
| `Google-Extended` | Google | Training (Gemini models & Vertex AI) | Yes | Controls Gemini training/fine-tuning; does **not** remove pages from Google Search or AI Overviews |
| `Google-Agent` | Google | Live Fetch / Action (Project Mariner & Gemini agents) | Browser / Session | Identifies browser-mediated agentic sessions (often paired with Web Bot Auth signatures) |
| `bingbot` | Microsoft | Index (Bing Search & Copilot grounding) | Yes | Reverse DNS (`*.search.msn.com`) & published IP list; powers Copilot citations |
| `Applebot-Extended` | Apple | Training (Apple Intelligence) | Yes | Separate from `Applebot` search index; opt-out of Apple foundation model training |
| `Amazonbot` | Amazon | Index & Training (Alexa / Rufus / Q) | Yes | Reverse DNS (`*.crawl.amazonbot.amazon`); supports standard `robots.txt` rules |
| `CCBot` | Common Crawl | Training (open web corpus) | Yes | Public dataset widely ingested by third-party LLM pre-training pipelines |
| `Meta-ExternalAgent` | Meta | Training & Index (Meta AI) | Yes | Matches case-insensitively (`meta-externalagent`); used for Meta AI training and retrieval |

### Copy-Paste `robots.txt`: Allow AI Search & User Agents, Block Model Training

    # 1. Allow AI search indexers & live user-triggered fetchers
    User-agent: OAI-SearchBot
    User-agent: ChatGPT-User
    User-agent: OAI-AdsBot
    User-agent: Claude-SearchBot
    User-agent: Claude-User
    User-agent: PerplexityBot
    User-agent: Perplexity-User
    User-agent: Googlebot
    User-agent: Google-Agent
    User-agent: bingbot
    Allow: /

    # 2. Block foundation-model training crawlers
    User-agent: GPTBot
    User-agent: ClaudeBot
    User-agent: Google-Extended
    User-agent: Applebot-Extended
    User-agent: Amazonbot
    User-agent: CCBot
    User-agent: Meta-ExternalAgent
    Disallow: /

    # 3. Default policy & discovery pointers
    User-agent: *
    Content-Signal: search=yes, ai-input=yes, ai-train=no
    Allow: /
    Sitemap: https://example.com/sitemap.xml
    Agentmap: https://example.com/.well-known/ard.json

# Knowledge Graph

* **Related standards**: [Content Signals Policy](/okf/protocols/cloudflare-content-signals-policy.md)

[^ref-1]: https://developers.openai.com/api/docs/bots (https://developers.openai.com/api/docs/bots)
[^ref-2]: https://support.claude.com/en/articles/8896518 (https://support.claude.com/en/articles/8896518)
[^ref-3]: https://docs.perplexity.ai/guides/bots (https://docs.perplexity.ai/guides/bots)