# AI Crawler User-Agent Tokens in robots.txt vs Content Signals Policy

The map lists them as complementary: sites and agents commonly implement both. AI Crawler User-Agent Tokens in robots.txt is a prerequisite of Content Signals Policy.

[3 Permit](/category/permit/) [AI Crawler User-Agent Tokens in robots.txt](/protocols/ai-crawler-tokens-robots/) · [3 Permit](/category/permit/) [Content Signals Policy](/protocols/cloudflare-content-signals-policy/)

|  | [AI Crawler User-Agent Tokens in robots.txt](/protocols/ai-crawler-tokens-robots/) | [Content Signals Policy](/protocols/cloudflare-content-signals-policy/) |
| --- | --- | --- |
| Lifecycle step | 3 Permit: What may the agent do here? | 3 Permit: What may the agent do here? |
| Answers | Which named AI crawler or agent user-agent tokens may a website allow or disallow in robots.txt, and what does each token actually control (search indexing, AI training, or live user-triggered fetches)? | May content fetched under a robots.txt allow rule also be used for search indexing, as live input to an AI answer, or for AI model training? |
| Governed by | No single governing body; each vendor unilaterally defines and documents its own token(s); RFC 9309 (IETF, Standards Track) governs the underlying robots.txt syntax all of them use | Cloudflare, developed with publisher/platform partners named in the announcement (e.g. Condé Nast, News/Media Alliance, Reddit, Pinterest, Quora, Yahoo) |
| Status (issuer's words) | A convention, not a formal spec; each vendor's documentation is normative only for that vendor's own crawler | Vendor-published policy convention layered on robots.txt; not submitted to a standards body as of this verification |
| Version | Ongoing; tokens and their scopes are updated by each vendor independently | Announced 2025-09-24 (2025-09-24) |
| Form / transport | well-known file | well-known file |
| Found at | /robots.txt | /robots.txt |
| Implementation | multi-implementation | multi-implementation |
| Evidence | 5 verified, 1 reported | 5 verified |
| Primary use case | Control AI crawling and training use | Control AI crawling and training use |
| Last verified | 2026-10-23 (current) | 2026-10-23 (current) |
| Specification | [https://www.rfc-editor.org/rfc/rfc9309.html](https://www.rfc-editor.org/rfc/rfc9309.html) | [https://blog.cloudflare.com/content-signals-policy/](https://blog.cloudflare.com/content-signals-policy/) |

## When to use which

- Choose AI Crawler User-Agent Tokens in robots.txt when your question is “Which named AI crawler or agent user-agent tokens may a website allow or disallow in robots.txt, and what does each token actually control (search indexing, AI training, or live user-triggered fetches)?” and you accept its maturity: status "A convention, not a formal spec; each vendor's documentation is normative only for that vendor's own crawler", implementation multi-implementation.
- Choose Content Signals Policy when your question is “May content fetched under a robots.txt allow rule also be used for search indexing, as live input to an AI answer, or for AI model training?” and you accept its maturity: status "Vendor-published policy convention layered on robots.txt; not submitted to a standards body as of this verification", implementation multi-implementation.

Guidance is derived from the two records (question, status, implementation, prerequisites, pairings), not from opinion. Each record's page carries its numbered references.

## Read the full articles

- [AI Crawler User-Agent Tokens in robots.txt](/protocols/ai-crawler-tokens-robots/): Which named AI crawler or agent user-agent tokens may a website allow or disallow in robots.txt, and what does each token actually control (search indexing, AI training, or live user-triggered fetches)?
- [Content Signals Policy](/protocols/cloudflare-content-signals-policy/): May content fetched under a robots.txt allow rule also be used for search indexing, as live input to an AI answer, or for AI model training?

Ask the MCP server to compare any two: the compare-protocols prompt, or get on both ids. [Connect](/connect/).

[ai-crawler-tokens-robots.json](/data/protocols/ai-crawler-tokens-robots.json) · [cloudflare-content-signals-policy.json](/data/protocols/cloudflare-content-signals-policy.json) · [Markdown](/compare/ai-crawler-tokens-robots-vs-cloudflare-content-signals-policy/index.md)