AI Crawler User-Agent Tokens in robots.txt vs agent-permissions.json
The map lists them as complementary: sites and agents commonly implement both.
| AI Crawler User-Agent Tokens in robots.txt | agent-permissions.json | |
|---|---|---|
| Lifecycle step | 3 Permit: What may the agent do here? | 3 Permit: What may the agent do here? |
| Answers | Which named AI crawler or agent user-agent tokens may a website allow or disallow in robots.txt, and what does each token actually control (search indexing, AI training, or live user-triggered fetches)? | Which page-level interactions (clicking, form submission, file upload, media control, script execution) may an agent perform while driving a web page's UI, as opposed to just fetching it? |
| Governed by | No single governing body; each vendor unilaterally defines and documents its own token(s); RFC 9309 (IETF, Standards Track) governs the underlying robots.txt syntax all of them use | Lightweight Agent Standards Working Group (LAS-WG), an open community project hosted on GitHub at las-wg/agent-permissions.json; no formal standards-body affiliation found |
| Status (issuer's words) | A convention, not a formal spec; each vendor's documentation is normative only for that vendor's own crawler | v1.0.0, complete JSON Schema published; described by the working group's own arXiv paper as a proposal, not an adopted standard |
| Phase | living | candidate |
| Version | Ongoing; tokens and their scopes are updated by each vendor independently | 1.0.0 (2026-07-17) |
| Form / transport | well-known file | well-known file |
| Found at | /robots.txt | /.well-known/agent-permissions.json |
| Adoption | observed | unmeasured |
| Evidence | 4 verified, 1 reported | 2 verified, 1 reported |
| Primary use case | Control AI crawling and training use | Let agents act on my website |
| Last verified | 2026-10-16 (current) | 2026-10-19 (current) |
| Specification | https://www.rfc-editor.org/rfc/rfc9309.html | https://github.com/las-wg/agent-permissions.json |
When to use which
- Choose AI Crawler User-Agent Tokens in robots.txt when your question is “Which named AI crawler or agent user-agent tokens may a website allow or disallow in robots.txt, and what does each token actually control (search indexing, AI training, or live user-triggered fetches)?” and you accept its maturity: phase living, adoption observed.
- Choose agent-permissions.json when your question is “Which page-level interactions (clicking, form submission, file upload, media control, script execution) may an agent perform while driving a web page's UI, as opposed to just fetching it?” and you accept its maturity: phase candidate, adoption unmeasured.
Read the full articles
- AI Crawler User-Agent Tokens in robots.txt: Which named AI crawler or agent user-agent tokens may a website allow or disallow in robots.txt, and what does each token actually control (search indexing, AI training, or live user-triggered fetches)?
- agent-permissions.json: Which page-level interactions (clicking, form submission, file upload, media control, script execution) may an agent perform while driving a web page's UI, as opposed to just fetching it?
ai-crawler-tokens-robots.json · agent-permissions-json.json · Markdown