# AI crawler tokens

[3 · Permit](/category/permit/)

| AI crawler tokens |
| --- |
| Category | [Permit](/category/permit/) |
| Also known as | crawler user-agent tokens, robots.txt AI bots |
| Related | [Content Signals Policy](/protocols/cloudflare-content-signals-policy/) |

[Permit](/category/permit/) AI crawler tokens are the User-agent names AI companies document for use in robots.txt under RFC 9309, split by purpose rather than treated as one identity. OpenAI documents OAI-SearchBot (search inclusion in ChatGPT), GPTBot (training data crawling), OAI-AdsBot (ad-landing-page checks) and ChatGPT-User (live fetches triggered by a user's own request, which is not governed by the search or training tokens). Anthropic and Perplexity publish an equivalent three-way split for their own products. Disallowing one token does not affect the others: a site can, for example, permit search indexing while refusing to have its content used for model training. ChatGPT-User- and Perplexity-User-class tokens are user-triggered fetches, not automated crawling, so robots.txt rules may not apply to them the way they do to a crawler. Used by: cloudflare-content-signals-policy.

## Instances

Not applicable.

## See also

- [Content Signals Policy](/protocols/cloudflare-content-signals-policy/)

## References

- [https://developers.openai.com/api/docs/bots](https://developers.openai.com/api/docs/bots) (2026-09-05) VERIFIED
- [https://support.claude.com/en/articles/8896518](https://support.claude.com/en/articles/8896518) (2026-04-07) VERIFIED
- [https://docs.perplexity.ai/guides/bots](https://docs.perplexity.ai/guides/bots) (2026-09-05) REPORTED

[JSON](/terms/ai-crawler-tokens/index.json) · [Markdown](/terms/ai-crawler-tokens/index.md)