{
 "url": "https://agenticweb.wiki/compare/ai-crawler-tokens-robots-vs-ai-txt/",
 "title": "AI Crawler User-Agent Tokens in robots.txt vs ai.txt",
 "description": "AI Crawler User-Agent Tokens in robots.txt vs ai.txt: The map lists them as complementary: sites and agents commonly implement both. Side-by-side step…",
 "markdown_url": "https://agenticweb.wiki/compare/ai-crawler-tokens-robots-vs-ai-txt/index.md",
 "markdown": "# AI Crawler User-Agent Tokens in robots.txt vs ai.txt\n\nThe map lists them as complementary: sites and agents commonly implement both.\n\n[3 Permit](/category/permit/) [AI Crawler User-Agent Tokens in robots.txt](/protocols/ai-crawler-tokens-robots/) · [3 Permit](/category/permit/) [ai.txt](/protocols/ai-txt/)\n\n|  | [AI Crawler User-Agent Tokens in robots.txt](/protocols/ai-crawler-tokens-robots/) | [ai.txt](/protocols/ai-txt/) |\n| --- | --- | --- |\n| Lifecycle step | 3 Permit: What may the agent do here? | 3 Permit: What may the agent do here? |\n| Answers | Which named AI crawler or agent user-agent tokens may a website allow or disallow in robots.txt, and what does each token actually control (search indexing, AI training, or live user-triggered fetches)? | May AI systems train on, scrape, index, or cache this site's content, and under what per-agent or per-path conditions? |\n| Governed by | No single governing body; each vendor unilaterally defines and documents its own token(s); RFC 9309 (IETF, Standards Track) governs the underlying robots.txt syntax all of them use | None; at least three independent proposals for the same policy-declaration purpose (an IETF Internet-Draft by 'car', kaylacar/ai-txt, and an arXiv DSL paper) plus a fourth, differently-scoped product (aitxt.ing) all use the same filename with no shared authority |\n| Status (issuer's words) | A convention, not a formal spec; each vendor's documentation is normative only for that vendor's own crawler | IETF Internet-Draft (draft-car-ai-txt-wellknown-00) presented as a formal well-known-URI registration proposal; kaylacar/ai-txt separately claims IANA filing (#76/#77, 'under review') |\n| Version | Ongoing; tokens and their scopes are updated by each vendor independently | draft-car-ai-txt-wellknown-00 |\n| Form / transport | well-known file | well-known file |\n| Found at | /robots.txt | /ai.txt or /.well-known/ai.txt (varies by proposal) |\n| Implementation | multi-implementation | prototype |\n| Evidence | 5 verified, 1 reported | 3 verified, 1 thin |\n| Primary use case | Control AI crawling and training use | Control AI crawling and training use |\n| Last verified | 2026-10-23 (current) | 2026-10-23 (current) |\n| Specification | [https://www.rfc-editor.org/rfc/rfc9309.html](https://www.rfc-editor.org/rfc/rfc9309.html) | [https://www.ietf.org/archive/id/draft-car-ai-txt-wellknown-00.html](https://www.ietf.org/archive/id/draft-car-ai-txt-wellknown-00.html) |\n\n## When to use which\n\n- Choose AI Crawler User-Agent Tokens in robots.txt when your question is “Which named AI crawler or agent user-agent tokens may a website allow or disallow in robots.txt, and what does each token actually control (search indexing, AI training, or live user-triggered fetches)?” and you accept its maturity: status \"A convention, not a formal spec; each vendor's documentation is normative only for that vendor's own crawler\", implementation multi-implementation.\n- Choose ai.txt when your question is “May AI systems train on, scrape, index, or cache this site's content, and under what per-agent or per-path conditions?” and you accept its maturity: status \"IETF Internet-Draft (draft-car-ai-txt-wellknown-00) presented as a formal well-known-URI registration proposal; kaylacar/ai-txt separately claims IANA filing (#76/#77, 'under review')\", implementation prototype.\n\nGuidance is derived from the two records (question, status, implementation, prerequisites, pairings), not from opinion. Each record's page carries its numbered references.\n\n## Read the full articles\n\n- [AI Crawler User-Agent Tokens in robots.txt](/protocols/ai-crawler-tokens-robots/): Which named AI crawler or agent user-agent tokens may a website allow or disallow in robots.txt, and what does each token actually control (search indexing, AI training, or live user-triggered fetches)?\n- [ai.txt](/protocols/ai-txt/): May AI systems train on, scrape, index, or cache this site's content, and under what per-agent or per-path conditions?\n\nAsk the MCP server to compare any two: the compare-protocols prompt, or get on both ids. [Connect](/connect/).\n\n[ai-crawler-tokens-robots.json](/data/protocols/ai-crawler-tokens-robots.json) · [ai-txt.json](/data/protocols/ai-txt.json) · [Markdown](/compare/ai-crawler-tokens-robots-vs-ai-txt/index.md)\n"
}