# AI Crawler User-Agent Tokens in robots.txt vs Text and Data Mining Reservation Protocol (TDMRep)

The map lists them as complementary: sites and agents commonly implement both.

[3 Permit](/category/permit/) [AI Crawler User-Agent Tokens in robots.txt](/protocols/ai-crawler-tokens-robots/) · [3 Permit](/category/permit/) [Text and Data Mining Reservation Protocol](/protocols/tdmrep/)

|  | [AI Crawler User-Agent Tokens in robots.txt](/protocols/ai-crawler-tokens-robots/) | [Text and Data Mining Reservation Protocol](/protocols/tdmrep/) TDMRep |
| --- | --- | --- |
| Lifecycle step | 3 Permit: What may the agent do here? | 3 Permit: What may the agent do here? |
| Answers | Which named AI crawler or agent user-agent tokens may a website allow or disallow in robots.txt, and what does each token actually control (search indexing, AI training, or live user-triggered fetches)? | Has a rights holder reserved text-and-data-mining rights over this content, and what policy governs any permitted mining? |
| Governed by | No single governing body; each vendor unilaterally defines and documents its own token(s); RFC 9309 (IETF, Standards Track) governs the underlying robots.txt syntax all of them use | W3C Text and Data Mining Reservation Protocol Community Group |
| Status (issuer's words) | A convention, not a formal spec; each vendor's documentation is normative only for that vendor's own crawler | W3C Community Group Final Report; explicitly "not a W3C Standard nor is it on the W3C Standards Track" |
| Version | Ongoing; tokens and their scopes are updated by each vendor independently | CG-FINAL-tdmrep-20240510 (2024-05-10) |
| Form / transport | well-known file | well-known file |
| Found at | /robots.txt | /.well-known/tdmrep.json |
| Implementation | multi-implementation | shipping |
| Evidence | 5 verified, 1 reported | 2 verified |
| Primary use case | Control AI crawling and training use | Control AI crawling and training use |
| Last verified | 2026-10-23 (current) | 2026-10-23 (current) |
| Specification | [https://www.rfc-editor.org/rfc/rfc9309.html](https://www.rfc-editor.org/rfc/rfc9309.html) | [https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240510/](https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240510/) |

## When to use which

- Choose AI Crawler User-Agent Tokens in robots.txt when your question is “Which named AI crawler or agent user-agent tokens may a website allow or disallow in robots.txt, and what does each token actually control (search indexing, AI training, or live user-triggered fetches)?” and you accept its maturity: status "A convention, not a formal spec; each vendor's documentation is normative only for that vendor's own crawler", implementation multi-implementation.
- Choose Text and Data Mining Reservation Protocol when your question is “Has a rights holder reserved text-and-data-mining rights over this content, and what policy governs any permitted mining?” and you accept its maturity: status "W3C Community Group Final Report; explicitly "not a W3C Standard nor is it on the W3C Standards Track"", implementation shipping.

Guidance is derived from the two records (question, status, implementation, prerequisites, pairings), not from opinion. Each record's page carries its numbered references.

## Read the full articles

- [AI Crawler User-Agent Tokens in robots.txt](/protocols/ai-crawler-tokens-robots/): Which named AI crawler or agent user-agent tokens may a website allow or disallow in robots.txt, and what does each token actually control (search indexing, AI training, or live user-triggered fetches)?
- [Text and Data Mining Reservation Protocol](/protocols/tdmrep/): Has a rights holder reserved text-and-data-mining rights over this content, and what policy governs any permitted mining?

Ask the MCP server to compare any two: the compare-protocols prompt, or get on both ids. [Connect](/connect/).

[ai-crawler-tokens-robots.json](/data/protocols/ai-crawler-tokens-robots.json) · [tdmrep.json](/data/protocols/tdmrep.json) · [Markdown](/compare/ai-crawler-tokens-robots-vs-tdmrep/index.md)