AI Crawler User-Agent Tokens in robots.txt vs Text and Data Mining Reservation Protocol (TDMRep)
The map lists them as complementary: sites and agents commonly implement both.
| AI Crawler User-Agent Tokens in robots.txt | Text and Data Mining Reservation Protocol TDMRep | |
|---|---|---|
| Lifecycle step | 3 Permit: What may the agent do here? | 3 Permit: What may the agent do here? |
| Answers | Which named AI crawler or agent user-agent tokens may a website allow or disallow in robots.txt, and what does each token actually control (search indexing, AI training, or live user-triggered fetches)? | Has a rights holder reserved text-and-data-mining rights over this content, and what policy governs any permitted mining? |
| Governed by | No single governing body; each vendor unilaterally defines and documents its own token(s); RFC 9309 (IETF, Standards Track) governs the underlying robots.txt syntax all of them use | W3C Text and Data Mining Reservation Protocol Community Group |
| Status (issuer's words) | A convention, not a formal spec; each vendor's documentation is normative only for that vendor's own crawler | W3C Community Group Final Report; explicitly "not a W3C Standard nor is it on the W3C Standards Track" |
| Version | Ongoing; tokens and their scopes are updated by each vendor independently | CG-FINAL-tdmrep-20240510 (2024-05-10) |
| Form / transport | well-known file | well-known file |
| Found at | /robots.txt | /.well-known/tdmrep.json |
| Implementation | multi-implementation | shipping |
| Evidence | 5 verified, 1 reported | 2 verified |
| Primary use case | Control AI crawling and training use | Control AI crawling and training use |
| Last verified | 2026-10-23 (current) | 2026-10-23 (current) |
| Specification | https://www.rfc-editor.org/rfc/rfc9309.html | https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240510/ |
When to use which
- Choose AI Crawler User-Agent Tokens in robots.txt when your question is “Which named AI crawler or agent user-agent tokens may a website allow or disallow in robots.txt, and what does each token actually control (search indexing, AI training, or live user-triggered fetches)?” and you accept its maturity: status "A convention, not a formal spec; each vendor's documentation is normative only for that vendor's own crawler", implementation multi-implementation.
- Choose Text and Data Mining Reservation Protocol when your question is “Has a rights holder reserved text-and-data-mining rights over this content, and what policy governs any permitted mining?” and you accept its maturity: status "W3C Community Group Final Report; explicitly "not a W3C Standard nor is it on the W3C Standards Track"", implementation shipping.
Read the full articles
- AI Crawler User-Agent Tokens in robots.txt: Which named AI crawler or agent user-agent tokens may a website allow or disallow in robots.txt, and what does each token actually control (search indexing, AI training, or live user-triggered fetches)?
- Text and Data Mining Reservation Protocol: Has a rights holder reserved text-and-data-mining rights over this content, and what policy governs any permitted mining?