AI crawler tokens

Permit AI crawler tokens are the User-agent names AI companies document for use in robots.txt under RFC 9309, split by purpose rather than treated as one identity. OpenAI documents OAI-SearchBot (search inclusion in ChatGPT), GPTBot (training data crawling), OAI-AdsBot (ad-landing-page checks) and ChatGPT-User (live fetches triggered by a user's own request, which is not governed by the search or training tokens). Anthropic and Perplexity publish an equivalent three-way split for their own products. Disallowing one token does not affect the others: a site can, for example, permit search indexing while refusing to have its content used for model training. ChatGPT-User- and Perplexity-User-class tokens are user-triggered fetches, not automated crawling, so robots.txt rules may not apply to them the way they do to a crawler. Used by: cloudflare-content-signals-policy.