agenticweb.wiki

ai.txt

Last checked · 23 primary sources

This page is about ai.txt as an AI usage-permission and training opt-out file: first Spawning Inc.'s root-level /ai.txt file (30 May 2023), which borrows robots.txt syntax (User-agent: * with Allow: and Disallow: rules over file extensions) to express commercial text-and-data-mining permissions [1][2][3][4], and second Kayla Cardillo's individual IETF Internet-Draft draft-car-ai-txt-wellknown-00 (12 June 2026; GitHub repository kaylacar/ai-txt created 22 February 2026), which proposes /.well-known/ai.txt and /.well-known/ai.json for Training, Scraping, Indexing, Caching, licensing, attribution and per-agent rules and explicitly "positions itself as a successor declaration surface rather than a competing one" to Spawning's 2023 file [5][6]. Three other unrelated proposals share the ai.txt filename: (a) menro/ai.txt (18 May 2023), an unmaintained GitHub sketch for an LLM content feed ("In essence, AI.txt is more akin to RSS feeds than robots.txt.") [11]; (b) the May 2025 academic paper ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet (Li et al., arXiv:2505.07834, 2 May 2025), which defines an element-level DSL at /ai.txt (Disallow: Return, Disallow: Train) with natural-language instructions and XML compilation [12][13]; and (c) aitxt.ing (January 2026), a cascading Markdown context file for describing resources so AI assistants "stop hallucinating", which states: "This specification is about context and boundaries in prose. It makes no claim on the permission-signalling space." [14]. It is not AI crawler tokens in robots.txt, the vendor user-agent tokens (GPTBot, ClaudeBot, Google-Extended) used inside robots.txt; it is not W3C TDMRep (/.well-known/tdmrep.json), the IETF AIPREF specifications (AIPREF Vocabulary, AIPREF Content-Usage Attachment), Cloudflare's Content Signals Policy, or Really Simple Licensing (RSL); and it is not llms.txt or agents.txt.

Overview

ai.txt is the name of two related plain-text AI training-permission file formats—first Spawning Inc.'s root-level /ai.txt file launched on 30 May 2023 [1][2], and second Kayla Cardillo's individual IETF Internet-Draft draft-car-ai-txt-wellknown-00 (12 June 2026), which proposes /.well-known/ai.txt and /.well-known/ai.json as an expanded successor to Spawning's file [5][6]. Both aim to let a website declare whether its content may be used for AI model training without blocking search engine indexing in robots.txt [1][5]. Neither format is on a standards-body track, and no major AI crawler operator (OpenAI, Google, Anthropic, Perplexity) documents reading either file [8][23].

Spawning's /ai.txt (2023)

Kayla Cardillo's /.well-known/ai.txt and /.well-known/ai.json (draft-car-ai-txt-wellknown-00, 2026)

Who uses it

Spawning's own domain (https://spawning.ai/ai.txt) no longer serves an ai.txt file (it returns an HTML application shell). Searching GitHub code on 2026-10-30 found 150 repositories containing a Spawning AI ai.txt file and 7 containing a Spec-Version ai.txt file; probing candidate live origins directly (NoRedirect, HTTP 200) confirmed live deployments in both formats:

Who reads it

No major AI crawler operator (OpenAI, Google, Anthropic, Perplexity) documents reading /ai.txt or /.well-known/ai.txt; their crawler pages direct site owners to robots.txt [23]. Even Spawning's own open-source datadiligence Python library (datadiligence.rules) implements TDMRepHeader, XRobotsTagHeader, and SpawningAPI (https://opts-api.spawningaiapi.com/api/v2/query/urls/) rather than a standalone /ai.txt file rule [20]. The documented readers come from the two proposals' own authors:

Recent coverage

See also

References

  1. Cullen Miller (Spawning Blog), ai.txt: A new way for websites to set permissions for AI, 2023-05-30, checked 2026-10-30: "Ai.txt is a file placed at the root of a website, which selectively restricts or permits access to the site’s content and media–mirroring the widely adopted robots.txt standard. Unlike robots.txt, which is commonly read when a website is crawled, ai.txt is read when a site’s media is downloaded." "With ai.txt, website owners can control whether or not their work is used to train new AI models and can continue to use robots.txt to manage permissions for popular search engines." "Starting today, Spawning’s API will communicate the permissions set by ai.txt files to our growing network of AI researchers and partners, including Hugging Face and Stability AI ." "In contrast, ai.txt takes direct aim at the EU TDM Article 4 exception by explicitly providing a machine readable opt-out method for commercial text and data mining." "Spawning views ai.txt not as a unilateral standard, but rather as a simple and useful way to declare permissions for text and data mining." — https://spawning.substack.com/p/aitxt-a-new-way-for-websites-to-set
  2. Spawning Inc., GitHub - Spawning-Inc/spawning-wp-plugin: Building the consent layer for AI (repository created 2023-05-30 UTC), checked 2026-10-30: "spawning-wp is a WordPress plugin which allows users to decide which types of content on their site is permitted for use with AI training. The plugin provides users with simple toggle switches to opt-out specific content types which may be used to train AI, and also generates a file at the appropriate location which follows the selected rules." "2023-05-30T18:25:59Z" — https://github.com/Spawning-Inc/spawning-wp-plugin
  3. Spawning Inc., spawning-wp-plugin/spawning_ai/readme.txt at main · Spawning-Inc/spawning-wp-plugin, checked 2026-10-30: "This plugin creates a file named ai.txt with dynamic text based on user-selected checkboxes. The ai.txt file is deleted when the plugin is uninstalled." "The ai.txt file will automatically be built and saved to your root path (e.g. https://www.example.com/ai.txt)." — https://github.com/Spawning-Inc/spawning-wp-plugin/blob/main/spawning_ai/readme.txt
  4. Spawning Inc., spawning-wp-plugin/spawning_ai/config/ai_txt_options.json at main · Spawning-Inc/spawning-wp-plugin, checked 2026-10-30: ""label": "Images"" ""label": "Audio"" ""label": "Video"" ""label": "Text"" ""label": "Code"" ""userAgent": "\nUser-agent: \n"" ""globalDisallow": "Disallow: \nDisallow: /\n"" ""pre": "# Spawning AI\n"" ""copyright": "Copyright 2023 Spawning Inc."" — https://github.com/Spawning-Inc/spawning-wp-plugin/blob/main/spawning_ai/config/ai_txt_options.json
  5. Kayla Cardillo (Independent), ai-txt/ietf/draft-car-ai-txt-wellknown-00.md at master · kaylacar/ai-txt (AI.TXT: A Declaration File for AI Usage Preferences, Licensing, and Policy, draft-car-ai-txt-wellknown-00, 2026-06-12), checked 2026-10-30: "title: "AI.TXT: A Declaration File for AI Usage Preferences, Licensing, and Policy"" "docname: draft-car-ai-txt-wellknown-00" "date: 2026-06-12" "workgroup: Independent Submission" "name: Kayla Cardillo" "This document requests registration of two Well-Known URIs under the "/.well-known/" path: "ai.txt" and "ai.json". These URIs define a structured, machine-readable file in which a site operator can declare AI usage preferences (training, scraping, indexing, caching), licensing terms, required attribution, and per-agent rules." "Spawning ai.txt (2023) {{SPAWNING-AITXT}}: : An earlier file at "/ai.txt" published by Spawning Inc. for text-and-data-mining opt-out, scoped narrowly to TDM permission per file pattern. The format defined in this document is a strict superset, covering training, scraping, indexing, caching, per-agent rules, licensing, and attribution. The present document acknowledges Spawning's prior use of the name and positions itself as a successor declaration surface rather than a competing one." "The "ai.txt" file MUST be served at: ~~~ https://example.com/.well-known/ai.txt ~~~" "The file MUST be served with Content-Type "text/plain; charset=utf-8"." "These fields declare site-wide defaults. Each accepts "allow" or "deny". The value "conditional" is valid only for the Training field, where it activates the per-path rules defined in the Training Path Fields section; implementations encountering "conditional" on any other field SHOULD treat it as "deny"." "Training (OPTIONAL, default "deny"):" "Scraping (OPTIONAL, default "allow"):" "Indexing (OPTIONAL, default "allow"):" "Caching (OPTIONAL, default "allow"):" "When Training is "conditional", these fields specify per-path rules: Training-Allow (OPTIONAL): : Glob pattern for paths where training is permitted. Training-Deny (OPTIONAL): : Glob pattern for paths where training is denied." "Training-License (OPTIONAL): : SPDX license identifier {{SPDX}} for AI training use (e.g., "CC-BY-4.0")." "Agent blocks declare per-agent policy overrides. The wildcard "*" sets the default for all agents." "in the JSON form, the "specVersion" member, the "policies" member (with all four of its "training", "scraping", "indexing", and "caching" members), and the "agents" member are REQUIRED." ""ai.txt" is advisory. It declares the site owner's policy. Compliance is expected in good faith but is not enforced by the file itself." — https://github.com/kaylacar/ai-txt/blob/master/ietf/draft-car-ai-txt-wellknown-00.md
  6. Kayla Cardillo, GitHub - kaylacar/ai-txt: ai.txt — open standard proposal for declaring a website's AI policy (created 2026-02-22 UTC, pushed 2026-07-18 UTC), checked 2026-10-30: "IETF Internet-Draft: draft-car-ai-txt-wellknown-00 — published on the Datatracker as an individual submission (under review, not adopted; expires 2026-12-14). Local source: ietf/draft-car-ai-txt-wellknown-00.md." "IANA Well-Known URI registrations: ai.txt (#76) and ai.json (#77) — filed, under review." "npx @ai-txt/core check https://example.com --agent ClaudeBot" — https://github.com/kaylacar/ai-txt
  7. Kayla Cardillo, ai-txt/SPEC.md at master · kaylacar/ai-txt (ai.txt Specification, Version 1.0), checked 2026-10-30: "ai.txt Specification" "Version: 1.0" — https://github.com/kaylacar/ai-txt/blob/master/SPEC.md
  8. Mark Nottingham (@mnot, IANA Designated Expert) and Kayla Cardillo (@kaylacar), Registration request: ai.txt · Issue #76 · protocol-registries/well-known-uris (opened 2026-02-22 UTC, closed 2026-03-16 UTC), checked 2026-10-30: "The name you have chosen is very generic, but the specification you refer to does not have broad community discussion or implementation. Furthermore, the specification is in a private GitHub repository, which is not suitable for registration due to its lack of stability." "Accordingly, it's not suitable for registration at this time." "2026-03-16T04:27:22Z" — https://github.com/protocol-registries/well-known-uris/issues/76
  9. Mark Nottingham (@mnot, IANA Designated Expert), Registration request: ai.json · Issue #77 · protocol-registries/well-known-uris (opened 2026-02-22 UTC, closed 2026-03-16 UTC), checked 2026-10-30: "Accordingly, it's not suitable for registration at this time. I would suggest you either: 1. Re-submit with a more suitably specific name (e.g., kaylacar-ai.json) and publish a specification in a more durable way (e.g., as an Independent Stream RFC, through W3C or similar processes), or 2. Propose your specification for adoption by the IETF" "2026-03-16T04:27:41Z" — https://github.com/protocol-registries/well-known-uris/issues/77
  10. IANA, Well-Known URIs registry CSV (neither ai.txt nor ai.json is registered), checked 2026-10-30: "URI Suffix,Reference,Status,Change Controller,Related Information,Date Registered,Date Modified" — https://www.iana.org/assignments/well-known-uris/well-known-uris-1.csv
  11. menro, GitHub - menro/ai.txt: Proposed Standard for AI.txt (created 2023-05-18 UTC, pushed 2023-05-19 UTC), checked 2026-10-30: "AI.txt: A Guide for Preparing Website Content for Large Language Models" "In essence, AI.txt is more akin to RSS feeds than robots.txt." "2023-05-18T04:41:11Z" — https://github.com/menro/ai.txt
  12. Yuekang Li et al., [2505.07834] ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet, submitted 2025-05-02, checked 2026-10-30: "Title: ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet" "[Submitted on 2 May 2025]" "Furthermore, we propose two compliance mechanisms: XML-based programmatic enforcement and natural language prompt integration" — https://arxiv.org/abs/2505.07834
  13. Yuekang Li et al., ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet (arXiv HTML v1), 2025-05-02, checked 2026-10-30: "Disallow: Return" "Disallow: Train" — https://arxiv.org/html/2505.07834v1
  14. aitxt.ing, ai.txt — Universal Standard for AI Context, checked 2026-10-30: "ai.txt is a universal standard for describing any resource in plain text, enabling AI agents to discover context, learn facts directly, and stop hallucinating." "Prior Art Several other files share the name or the neighbourhood. They solve different problems and coexist with this one: Spawning ai.txt (2023) — a robots.txt-style file for AI training consent. Permissions, not context. IETF draft-car-ai-txt-wellknown (2026) — an individual Internet-Draft registering /.well-known/ai.txt and /.well-known/ai.json : key/value declarations of training, scraping, indexing and licensing policy, positioned as the successor to Spawning's file." "This specification is about context and boundaries in prose. It makes no claim on the permission-signalling space." — https://aitxt.ing/
  15. WordPress Openverse API, /ai.txt on api.openverse.org, checked 2026-10-30: "# Spawning AI # Prevent datasets from using the following file types User-Agent: Disallow: .txt" — https://api.openverse.org/ai.txt
  16. Republik, /ai.txt on www.republik.ch, checked 2026-10-30: "# Spawning AI # Prevent datasets from using the following file types User-Agent: Disallow: / Disallow: " — https://www.republik.ch/ai.txt
  17. MLTSHP, /ai.txt on mltshp.com, checked 2026-10-30: "# Spawning AI # Prevent datasets from using the following file types User-Agent: Disallow: .txt" — https://mltshp.com/ai.txt
  18. makigas, /ai.txt on www.makigas.es, checked 2026-10-30: "# Spawning AI # Prevent datasets from using the following file types User-Agent: Disallow: .txt" — https://www.makigas.es/ai.txt
  19. thayn.me, /.well-known/ai.txt, checked 2026-10-30: "# ai.txt — AI usage policy for thayn.me # IETF draft: https://datatracker.ietf.org/doc/draft-car-ai-txt-wellknown/" "Spec-Version: 1.0 Site-Name: thayn.me Site-URL: https://thayn.me" — https://thayn.me/.well-known/ai.txt
  20. Spawning Inc., datadiligence.rules package — datadiligence 0.1.7.post1.dev1+gea09739 documentation, checked 2026-10-30: "This module wraps HTTP calls to the Spawning AI API. See Spawning API documentation here: https://opts-api.spawningaiapi.com/docs ." — https://datadiligence.readthedocs.io/en/latest/api/datadiligence.rules.html
  21. ZenphotoCMS User Guide, Protecting your site from AI scraping, 2024-03-26, checked 2026-10-30: "There is also the Initiative Spawning that aims to create tools to provide a way to block or consent AI access by opting out. There you can add yourself to a registry and create a specific ai.txt which is similar to robots.txt: https://site.spawning.ai/spawning-ai-txt#ai-text-generator . Sadly it is more a concept of an opt-out infrastructure and so far only two AI companies seems to follow that registry." — https://www.zenphoto.org/news/protecting-your-site-from-ai-scraping/
  22. European Commission (DG CONNECT / AI Office), Stakeholder consultation on AI and copyright compliance, last updated 2025-12-01, checked 2026-10-30: "In addition to robots.txt, the study identifies seven other technical solutions (TDM reservation protocol (TDMRep); C2PA TDM Assertions; AI.txt; Do not train registry (Spawning AI); JPEG Trust core foundation v2; TDM.ai protocol (Liccium); Open Rights Data Exchange (Valunode)." — https://digital-strategy.ec.europa.eu/en/faqs/stakeholder-consultation-ai-and-copyright-compliance
  23. David (ContextBolt), ai.txt: The Permissions File That Is Not llms.txt, published 2026-09-18, updated 2026-10-22, checked 2026-10-30: "ai.txt is a training opt-out, not a content map. The file on real sites comes from Spawning’s 2023 generator. It borrows robots.txt syntax and applies it to file types, so a line like Disallow: *.jpg means “do not train on my images”." "Five specs share the name. Spawning’s opt-out, a 2023 GitHub proposal, a May 2025 arXiv paper, a 2026 anti-hallucination format at aitxt.ing, and a June 2026 individual IETF draft. None is a standard." "No major crawler documents reading it. OpenAI, Google, Anthropic and Perplexity all point you at robots.txt. Spawning relays the file to its own partners through its API, and that is the extent of it." — https://contextbolt.com/blog/ai-txt/

JSON · Markdown · OKF v0.2