--- type: Agentic Web Standard title: "ai.txt" description: "May AI systems train on, scrape, index, or cache this site's content, and under what media-type, per-path, or per-agent rules?" resource: "https://github.com/kaylacar/ai-txt/blob/master/ietf/draft-car-ai-txt-wellknown-00.md" tags: ["permit", "well-known-file", "community", "control-ai-crawling"] status: draft generated: { by: agenticweb-builder/1.0, at: 2026-10-30T00:00:00Z } verified: { by: process:agenticweb-engine, at: 2026-10-30T00:00:00Z } stale_after: 2026-11-29T00:00:00Z sources: - id: ref-1 resource: "https://spawning.substack.com/p/aitxt-a-new-way-for-websites-to-set" title: "Cullen Miller (Spawning Blog), ai.txt: A new way for websites to set permissions for AI, 2023-05-30, checked 2026-10-30: \"Ai.txt is a file placed at the root of a website, which selectively restricts or permits access to the site’s content and media–mirroring the widely adopted robots.txt standard. Unlike robots.txt, which is commonly read when a website is crawled, ai.txt is read when a site’s media is downloaded.\" \"With ai.txt, website owners can control whether or not their work is used to train new AI models and can continue to use robots.txt to manage permissions for popular search engines.\" \"Starting today, Spawning’s API will communicate the permissions set by ai.txt files to our growing network of AI researchers and partners, including Hugging Face and Stability AI .\" \"In contrast, ai.txt takes direct aim at the EU TDM Article 4 exception by explicitly providing a machine readable opt-out method for commercial text and data mining.\" \"Spawning views ai.txt not as a unilateral standard, but rather as a simple and useful way to declare permissions for text and data mining.\"" last_modified: 2023-05-30T00:00:00Z - id: ref-2 resource: "https://github.com/Spawning-Inc/spawning-wp-plugin" title: "Spawning Inc., GitHub - Spawning-Inc/spawning-wp-plugin: Building the consent layer for AI (repository created 2023-05-30 UTC), checked 2026-10-30: \"spawning-wp is a WordPress plugin which allows users to decide which types of content on their site is permitted for use with AI training. The plugin provides users with simple toggle switches to opt-out specific content types which may be used to train AI, and also generates a file at the appropriate location which follows the selected rules.\" \"2023-05-30T18:25:59Z\"" last_modified: 2023-05-30T00:00:00Z - id: ref-3 resource: "https://github.com/Spawning-Inc/spawning-wp-plugin/blob/main/spawning_ai/readme.txt" title: "Spawning Inc., spawning-wp-plugin/spawning_ai/readme.txt at main · Spawning-Inc/spawning-wp-plugin, checked 2026-10-30: \"This plugin creates a file named ai.txt with dynamic text based on user-selected checkboxes. The ai.txt file is deleted when the plugin is uninstalled.\" \"The ai.txt file will automatically be built and saved to your root path (e.g. https://www.example.com/ai.txt).\"" last_modified: 2023-05-30T00:00:00Z - id: ref-4 resource: "https://github.com/Spawning-Inc/spawning-wp-plugin/blob/main/spawning_ai/config/ai_txt_options.json" title: "Spawning Inc., spawning-wp-plugin/spawning_ai/config/ai_txt_options.json at main · Spawning-Inc/spawning-wp-plugin, checked 2026-10-30: \"\"label\": \"Images\"\" \"\"label\": \"Audio\"\" \"\"label\": \"Video\"\" \"\"label\": \"Text\"\" \"\"label\": \"Code\"\" \"\"userAgent\": \"\\nUser-agent: *\\n\"\" \"\"globalDisallow\": \"Disallow: *\\nDisallow: /\\n\"\" \"\"pre\": \"# Spawning AI\\n\"\" \"\"copyright\": \"Copyright 2023 Spawning Inc.\"\"" last_modified: 2023-05-30T00:00:00Z - id: ref-5 resource: "https://github.com/kaylacar/ai-txt/blob/master/ietf/draft-car-ai-txt-wellknown-00.md" title: "Kayla Cardillo (Independent), ai-txt/ietf/draft-car-ai-txt-wellknown-00.md at master · kaylacar/ai-txt (AI.TXT: A Declaration File for AI Usage Preferences, Licensing, and Policy, draft-car-ai-txt-wellknown-00, 2026-06-12), checked 2026-10-30: \"title: \"AI.TXT: A Declaration File for AI Usage Preferences, Licensing, and Policy\"\" \"docname: draft-car-ai-txt-wellknown-00\" \"date: 2026-06-12\" \"workgroup: Independent Submission\" \"name: Kayla Cardillo\" \"This document requests registration of two Well-Known URIs under the \"/.well-known/\" path: \"ai.txt\" and \"ai.json\". These URIs define a structured, machine-readable file in which a site operator can declare AI usage preferences (training, scraping, indexing, caching), licensing terms, required attribution, and per-agent rules.\" \"Spawning ai.txt (2023) {{SPAWNING-AITXT}}: : An earlier file at \"/ai.txt\" published by Spawning Inc. for text-and-data-mining opt-out, scoped narrowly to TDM permission per file pattern. The format defined in this document is a strict superset, covering training, scraping, indexing, caching, per-agent rules, licensing, and attribution. The present document acknowledges Spawning's prior use of the name and positions itself as a successor declaration surface rather than a competing one.\" \"The \"ai.txt\" file MUST be served at: ~~~ https://example.com/.well-known/ai.txt ~~~\" \"The file MUST be served with Content-Type \"text/plain; charset=utf-8\".\" \"These fields declare site-wide defaults. Each accepts \"allow\" or \"deny\". The value \"conditional\" is valid only for the Training field, where it activates the per-path rules defined in the Training Path Fields section; implementations encountering \"conditional\" on any other field SHOULD treat it as \"deny\".\" \"Training (OPTIONAL, default \"deny\"):\" \"Scraping (OPTIONAL, default \"allow\"):\" \"Indexing (OPTIONAL, default \"allow\"):\" \"Caching (OPTIONAL, default \"allow\"):\" \"When Training is \"conditional\", these fields specify per-path rules: Training-Allow (OPTIONAL): : Glob pattern for paths where training is permitted. Training-Deny (OPTIONAL): : Glob pattern for paths where training is denied.\" \"Training-License (OPTIONAL): : SPDX license identifier {{SPDX}} for AI training use (e.g., \"CC-BY-4.0\").\" \"Agent blocks declare per-agent policy overrides. The wildcard \"*\" sets the default for all agents.\" \"in the JSON form, the \"specVersion\" member, the \"policies\" member (with all four of its \"training\", \"scraping\", \"indexing\", and \"caching\" members), and the \"agents\" member are REQUIRED.\" \"\"ai.txt\" is advisory. It declares the site owner's policy. Compliance is expected in good faith but is not enforced by the file itself.\"" last_modified: 2026-06-12T00:00:00Z - id: ref-6 resource: "https://github.com/kaylacar/ai-txt" title: "Kayla Cardillo, GitHub - kaylacar/ai-txt: ai.txt — open standard proposal for declaring a website's AI policy (created 2026-02-22 UTC, pushed 2026-07-18 UTC), checked 2026-10-30: \"IETF Internet-Draft: draft-car-ai-txt-wellknown-00 — published on the Datatracker as an individual submission (under review, not adopted; expires 2026-12-14). Local source: ietf/draft-car-ai-txt-wellknown-00.md.\" \"IANA Well-Known URI registrations: ai.txt (#76) and ai.json (#77) — filed, under review.\" \"npx @ai-txt/core check https://example.com --agent ClaudeBot\"" last_modified: 2026-07-18T00:00:00Z - id: ref-7 resource: "https://github.com/kaylacar/ai-txt/blob/master/SPEC.md" title: "Kayla Cardillo, ai-txt/SPEC.md at master · kaylacar/ai-txt (ai.txt Specification, Version 1.0), checked 2026-10-30: \"ai.txt Specification\" \"Version: 1.0\"" last_modified: 2026-07-18T00:00:00Z - id: ref-8 resource: "https://github.com/protocol-registries/well-known-uris/issues/76" title: "Mark Nottingham (@mnot, IANA Designated Expert) and Kayla Cardillo (@kaylacar), Registration request: ai.txt · Issue #76 · protocol-registries/well-known-uris (opened 2026-02-22 UTC, closed 2026-03-16 UTC), checked 2026-10-30: \"The name you have chosen is very generic, but the specification you refer to does not have broad community discussion or implementation. Furthermore, the specification is in a private GitHub repository, which is not suitable for registration due to its lack of stability.\" \"Accordingly, it's not suitable for registration at this time.\" \"2026-03-16T04:27:22Z\"" last_modified: 2026-03-16T00:00:00Z - id: ref-9 resource: "https://github.com/protocol-registries/well-known-uris/issues/77" title: "Mark Nottingham (@mnot, IANA Designated Expert), Registration request: ai.json · Issue #77 · protocol-registries/well-known-uris (opened 2026-02-22 UTC, closed 2026-03-16 UTC), checked 2026-10-30: \"Accordingly, it's not suitable for registration at this time. I would suggest you either: 1. Re-submit with a more suitably specific name (e.g., kaylacar-ai.json) and publish a specification in a more durable way (e.g., as an Independent Stream RFC, through W3C or similar processes), or 2. Propose your specification for adoption by the IETF\" \"2026-03-16T04:27:41Z\"" last_modified: 2026-03-16T00:00:00Z - id: ref-10 resource: "https://www.iana.org/assignments/well-known-uris/well-known-uris-1.csv" title: "IANA, Well-Known URIs registry CSV (neither ai.txt nor ai.json is registered), checked 2026-10-30: \"URI Suffix,Reference,Status,Change Controller,Related Information,Date Registered,Date Modified\"" last_modified: 2026-10-30T00:00:00Z - id: ref-11 resource: "https://github.com/menro/ai.txt" title: "menro, GitHub - menro/ai.txt: Proposed Standard for AI.txt (created 2023-05-18 UTC, pushed 2023-05-19 UTC), checked 2026-10-30: \"AI.txt: A Guide for Preparing Website Content for Large Language Models\" \"In essence, AI.txt is more akin to RSS feeds than robots.txt.\" \"2023-05-18T04:41:11Z\"" last_modified: 2023-05-19T00:00:00Z - id: ref-12 resource: "https://arxiv.org/abs/2505.07834" title: "Yuekang Li et al., [2505.07834] ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet, submitted 2025-05-02, checked 2026-10-30: \"Title: ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet\" \"[Submitted on 2 May 2025]\" \"Furthermore, we propose two compliance mechanisms: XML-based programmatic enforcement and natural language prompt integration\"" last_modified: 2025-05-02T00:00:00Z - id: ref-13 resource: "https://arxiv.org/html/2505.07834v1" title: "Yuekang Li et al., ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet (arXiv HTML v1), 2025-05-02, checked 2026-10-30: \"Disallow: Return\" \"Disallow: Train\"" last_modified: 2025-05-02T00:00:00Z - id: ref-14 resource: "https://aitxt.ing/" title: "aitxt.ing, ai.txt — Universal Standard for AI Context, checked 2026-10-30: \"ai.txt is a universal standard for describing any resource in plain text, enabling AI agents to discover context, learn facts directly, and stop hallucinating.\" \"Prior Art Several other files share the name or the neighbourhood. They solve different problems and coexist with this one: Spawning ai.txt (2023) — a robots.txt-style file for AI training consent. Permissions, not context. IETF draft-car-ai-txt-wellknown (2026) — an individual Internet-Draft registering /.well-known/ai.txt and /.well-known/ai.json : key/value declarations of training, scraping, indexing and licensing policy, positioned as the successor to Spawning's file.\" \"This specification is about context and boundaries in prose. It makes no claim on the permission-signalling space.\"" last_modified: 2026-01-14T00:00:00Z - id: ref-15 resource: "https://api.openverse.org/ai.txt" title: "WordPress Openverse API, /ai.txt on api.openverse.org, checked 2026-10-30: \"# Spawning AI # Prevent datasets from using the following file types User-Agent: * Disallow: *.txt\"" last_modified: 2026-10-30T00:00:00Z - id: ref-16 resource: "https://www.republik.ch/ai.txt" title: "Republik, /ai.txt on www.republik.ch, checked 2026-10-30: \"# Spawning AI # Prevent datasets from using the following file types User-Agent: * Disallow: / Disallow: *\"" last_modified: 2026-10-30T00:00:00Z - id: ref-17 resource: "https://mltshp.com/ai.txt" title: "MLTSHP, /ai.txt on mltshp.com, checked 2026-10-30: \"# Spawning AI # Prevent datasets from using the following file types User-Agent: * Disallow: *.txt\"" last_modified: 2026-10-30T00:00:00Z - id: ref-18 resource: "https://www.makigas.es/ai.txt" title: "makigas, /ai.txt on www.makigas.es, checked 2026-10-30: \"# Spawning AI # Prevent datasets from using the following file types User-Agent: * Disallow: *.txt\"" last_modified: 2026-10-30T00:00:00Z - id: ref-19 resource: "https://thayn.me/.well-known/ai.txt" title: "thayn.me, /.well-known/ai.txt, checked 2026-10-30: \"# ai.txt — AI usage policy for thayn.me # IETF draft: https://datatracker.ietf.org/doc/draft-car-ai-txt-wellknown/\" \"Spec-Version: 1.0 Site-Name: thayn.me Site-URL: https://thayn.me\"" last_modified: 2026-10-30T00:00:00Z - id: ref-20 resource: "https://datadiligence.readthedocs.io/en/latest/api/datadiligence.rules.html" title: "Spawning Inc., datadiligence.rules package — datadiligence 0.1.7.post1.dev1+gea09739 documentation, checked 2026-10-30: \"This module wraps HTTP calls to the Spawning AI API. See Spawning API documentation here: https://opts-api.spawningaiapi.com/docs .\"" last_modified: 2026-10-30T00:00:00Z - id: ref-21 resource: "https://www.zenphoto.org/news/protecting-your-site-from-ai-scraping/" title: "ZenphotoCMS User Guide, Protecting your site from AI scraping, 2024-03-26, checked 2026-10-30: \"There is also the Initiative Spawning that aims to create tools to provide a way to block or consent AI access by opting out. There you can add yourself to a registry and create a specific ai.txt which is similar to robots.txt: https://site.spawning.ai/spawning-ai-txt#ai-text-generator . Sadly it is more a concept of an opt-out infrastructure and so far only two AI companies seems to follow that registry.\"" last_modified: 2024-03-26T00:00:00Z - id: ref-22 resource: "https://digital-strategy.ec.europa.eu/en/faqs/stakeholder-consultation-ai-and-copyright-compliance" title: "European Commission (DG CONNECT / AI Office), Stakeholder consultation on AI and copyright compliance, last updated 2025-12-01, checked 2026-10-30: \"In addition to robots.txt, the study identifies seven other technical solutions (TDM reservation protocol (TDMRep); C2PA TDM Assertions; AI.txt; Do not train registry (Spawning AI); JPEG Trust core foundation v2; TDM.ai protocol (Liccium); Open Rights Data Exchange (Valunode).\"" last_modified: 2025-12-01T00:00:00Z - id: ref-23 resource: "https://contextbolt.com/blog/ai-txt/" title: "David (ContextBolt), ai.txt: The Permissions File That Is Not llms.txt, published 2026-09-18, updated 2026-10-22, checked 2026-10-30: \"ai.txt is a training opt-out, not a content map. The file on real sites comes from Spawning’s 2023 generator. It borrows robots.txt syntax and applies it to file types, so a line like Disallow: *.jpg means “do not train on my images”.\" \"Five specs share the name. Spawning’s opt-out, a 2023 GitHub proposal, a May 2025 arXiv paper, a 2026 anti-hallucination format at aitxt.ing, and a June 2026 individual IETF draft. None is a standard.\" \"No major crawler documents reading it. OpenAI, Google, Anthropic and Perplexity all point you at robots.txt. Spawning relays the file to its own partners through its API, and that is the extent of it.\"" last_modified: 2026-10-22T00:00:00Z --- # ai.txt ai.txt is an agentic-web standard (well-known file — /ai.txt) at step 3, Permit, governed by None across the filename; its status is "Spawning /ai.txt: shipping generator and WordPress plugin v1.0". It answers: "May AI systems train on, scrape, index, or cache this site's content, and under what media-type, per-path, or per-agent rules?" Implementation status is prototype, last verified 2026-10-30. # Overview `ai.txt` is the name of two related plain-text AI training-permission file formats—first Spawning Inc.'s root-level `/ai.txt` file launched on 30 May 2023 [^ref-1][^ref-2], and second Kayla Cardillo's individual IETF Internet-Draft `draft-car-ai-txt-wellknown-00` (12 June 2026), which proposes `/.well-known/ai.txt` and `/.well-known/ai.json` as an expanded successor to Spawning's file [^ref-5][^ref-6]. Both aim to let a website declare whether its content may be used for AI model training without blocking search engine indexing in `robots.txt` [^ref-1][^ref-5]. Neither format is on a standards-body track, and no major AI crawler operator (OpenAI, Google, Anthropic, Perplexity) documents reading either file [^ref-8][^ref-23]. ### Spawning's `/ai.txt` (2023) - **Purpose and download-time check.** Spawning announced `/ai.txt` on 30 May 2023 as a root-level file modeled on `robots.txt`: `"Ai.txt is a file placed at the root of a website, which selectively restricts or permits access to the site’s content and media–mirroring the widely adopted robots.txt standard. Unlike robots.txt, which is commonly read when a website is crawled, ai.txt is read when a site’s media is downloaded."` [^ref-1] Spawning framed the file as a machine-readable opt-out under Article 4 of the EU Digital Single Market Copyright Directive (2019/790): `"In contrast, ai.txt takes direct aim at the EU TDM Article 4 exception by explicitly providing a machine readable opt-out method for commercial text and data mining."` [^ref-1] - **File-extension syntax.** Spawning's generator and WordPress plugin (`Spawning-Inc/spawning-wp-plugin`, v1.0) write `/ai.txt` at the site root (`https://www.example.com/ai.txt`) [^ref-2][^ref-3]. Instead of path rules, the file opens with `# Spawning AI` and `User-agent: *` followed by `Allow:` or `Disallow:` lines for five media-type extension lists (`Images`, `Audio`, `Video`, `Text`, and `Code`, such as `Disallow: *.jpg`, `Disallow: *.mp3`, `Disallow: *.mp4`, `Disallow: *.txt`, `Disallow: *.py`), ending with `Disallow: *` and `Disallow: /` when all types are blocked (or `Allow: /` when `Text` is permitted) [^ref-4]. - **How permissions are relayed.** Rather than expecting every crawler to fetch `/ai.txt` directly, Spawning stated at launch that `"Spawning’s API will communicate the permissions set by ai.txt files to our growing network of AI researchers and partners, including Hugging Face and Stability AI ."` [^ref-1] The European Commission's 2025 GPAI Code of Practice copyright consultation FAQ lists `AI.txt` alongside `TDM reservation protocol (TDMRep)` and `Do not train registry (Spawning AI)` among the seven non-`robots.txt` technical opt-out solutions identified by the EUIPO study [^ref-22]. ### Kayla Cardillo's `/.well-known/ai.txt` and `/.well-known/ai.json` (`draft-car-ai-txt-wellknown-00`, 2026) - **Relationship to Spawning and AIPREF.** Published on 12 June 2026 by Kayla Cardillo (`kaylacar/ai-txt`, repository created 22 February 2026), `draft-car-ai-txt-wellknown-00` explicitly cites `Spawning ai.txt (2023)` (`[SPAWNING-AITXT]`) and states: `"The format defined in this document is a strict superset, covering training, scraping, indexing, caching, per-agent rules, licensing, and attribution. The present document acknowledges Spawning's prior use of the name and positions itself as a successor declaration surface rather than a competing one."` [^ref-5][^ref-6] It also positions itself as a third, single-file carriage surface alongside the IETF AIPREF working group's `robots.txt` and HTTP-header attachment drafts [^ref-5]. - **Discovery and key-value grammar.** Cardillo's draft requires `https://example.com/.well-known/ai.txt` (`Content-Type: text/plain; charset=utf-8`) and an optional JSON companion at `https://example.com/.well-known/ai.json` (`application/json; charset=utf-8`) [^ref-5]. A text file requires `Site-Name` and `Site-URL` and declares four site-wide policy fields—`Training` (`allow`, `deny`, or `conditional`, default `"deny"`), `Scraping` (default `"allow"`), `Indexing` (default `"allow"`), and `Caching` (default `"allow"`) [^ref-5]. When `Training` is `conditional`, `Training-Allow` and `Training-Deny` glob patterns set per-path rules, while `Training-License` carries an SPDX identifier (such as `CC-BY-4.0`) and `Training-Fee` links to commercial terms [^ref-5]. Indented `Agent:` blocks (such as `Agent: ClaudeBot` or `Agent: GPTBot`) override the site-wide policy and set advisory `Rate-Limit` values (`60/minute`), alongside `Attribution`, `AI-Disclosure`, and `Audit` fields [^ref-5]. In `ai.json`, `specVersion`, `policies` (with all four members explicit), and `agents` are required [^ref-5]. - **Advisory nature and IANA rejection.** The draft states that `""ai.txt" is advisory. It declares the site owner's policy. Compliance is expected in good faith but is not enforced by the file itself."` [^ref-5] Although `kaylacar/ai-txt` still describes IANA Well-Known URI filings `#76` (`ai.txt`) and `#77` (`ai.json`) as `"filed, under review"` [^ref-6], IANA Designated Expert Mark Nottingham (`@mnot`) rejected and closed both requests on 16 March 2026: `"The name you have chosen is very generic, but the specification you refer to does not have broad community discussion or implementation. Furthermore, the specification is in a private GitHub repository, which is not suitable for registration due to its lack of stability. Accordingly, it's not suitable for registration at this time."` [^ref-8][^ref-9] Neither `ai.txt` nor `ai.json` is listed in the IANA Well-Known URIs registry [^ref-10]. # Examples ```http GET {origin}/ai.txt # parse: 200 response with text/plain body starting with `# Spawning AI` / `User-Agent: *` and per-extension `Allow:` / `Disallow:` rules at `/ai.txt`, or `Spec-Version: 1.0` with `Training:` / `Scraping:` / `Indexing:` / `Caching:` fields at `/.well-known/ai.txt` # on absent: No /ai.txt or /.well-known/ai.txt file present; major AI crawlers consult /robots.txt instead ``` # Knowledge Graph * **Lifecycle Step**: [3 Permit](/category/permit/index.md) * **Pairs with**: [AI crawler tokens in robots.txt](/okf/protocols/ai-crawler-tokens-robots.md), [AIPREF Vocabulary (A Vocabulary For Expressing AI Usage Preferences)](/okf/protocols/aipref-vocabulary.md), [AIPREF Content-Usage Attachment](/okf/protocols/aipref-content-usage-attachment.md), [Content Signals Policy](/okf/protocols/cloudflare-content-signals-policy.md), [TDMRep (TDM Reservation Protocol)](/okf/protocols/tdmrep.md), [Really Simple Licensing](/okf/protocols/rsl.md) * **Compared in**: [robots.txt (Allow/Disallow) vs Content-Signal, AIPREF & TDMRep (classic vs agentic)](/compare/robots-txt-vs-content-signal/index.md), [AI crawler tokens in robots.txt vs ai.txt](/compare/ai-crawler-tokens-robots-vs-ai-txt/index.md) [^ref-1]: Cullen Miller (Spawning Blog), ai.txt: A new way for websites to set permissions for AI, 2023-05-30, checked 2026-10-30: "Ai.txt is a file placed at the root of a website, which selectively restricts or permits access to the site’s content and media–mirroring the widely adopted robots.txt standard. Unlike robots.txt, which is commonly read when a website is crawled, ai.txt is read when a site’s media is downloaded." "With ai.txt, website owners can control whether or not their work is used to train new AI models and can continue to use robots.txt to manage permissions for popular search engines." "Starting today, Spawning’s API will communicate the permissions set by ai.txt files to our growing network of AI researchers and partners, including Hugging Face and Stability AI ." "In contrast, ai.txt takes direct aim at the EU TDM Article 4 exception by explicitly providing a machine readable opt-out method for commercial text and data mining." "Spawning views ai.txt not as a unilateral standard, but rather as a simple and useful way to declare permissions for text and data mining." (https://spawning.substack.com/p/aitxt-a-new-way-for-websites-to-set) [^ref-2]: Spawning Inc., GitHub - Spawning-Inc/spawning-wp-plugin: Building the consent layer for AI (repository created 2023-05-30 UTC), checked 2026-10-30: "spawning-wp is a WordPress plugin which allows users to decide which types of content on their site is permitted for use with AI training. The plugin provides users with simple toggle switches to opt-out specific content types which may be used to train AI, and also generates a file at the appropriate location which follows the selected rules." "2023-05-30T18:25:59Z" (https://github.com/Spawning-Inc/spawning-wp-plugin) [^ref-3]: Spawning Inc., spawning-wp-plugin/spawning_ai/readme.txt at main · Spawning-Inc/spawning-wp-plugin, checked 2026-10-30: "This plugin creates a file named ai.txt with dynamic text based on user-selected checkboxes. The ai.txt file is deleted when the plugin is uninstalled." "The ai.txt file will automatically be built and saved to your root path (e.g. https://www.example.com/ai.txt)." (https://github.com/Spawning-Inc/spawning-wp-plugin/blob/main/spawning_ai/readme.txt) [^ref-4]: Spawning Inc., spawning-wp-plugin/spawning_ai/config/ai_txt_options.json at main · Spawning-Inc/spawning-wp-plugin, checked 2026-10-30: ""label": "Images"" ""label": "Audio"" ""label": "Video"" ""label": "Text"" ""label": "Code"" ""userAgent": "\nUser-agent: *\n"" ""globalDisallow": "Disallow: *\nDisallow: /\n"" ""pre": "# Spawning AI\n"" ""copyright": "Copyright 2023 Spawning Inc."" (https://github.com/Spawning-Inc/spawning-wp-plugin/blob/main/spawning_ai/config/ai_txt_options.json) [^ref-5]: Kayla Cardillo (Independent), ai-txt/ietf/draft-car-ai-txt-wellknown-00.md at master · kaylacar/ai-txt (AI.TXT: A Declaration File for AI Usage Preferences, Licensing, and Policy, draft-car-ai-txt-wellknown-00, 2026-06-12), checked 2026-10-30: "title: "AI.TXT: A Declaration File for AI Usage Preferences, Licensing, and Policy"" "docname: draft-car-ai-txt-wellknown-00" "date: 2026-06-12" "workgroup: Independent Submission" "name: Kayla Cardillo" "This document requests registration of two Well-Known URIs under the "/.well-known/" path: "ai.txt" and "ai.json". These URIs define a structured, machine-readable file in which a site operator can declare AI usage preferences (training, scraping, indexing, caching), licensing terms, required attribution, and per-agent rules." "Spawning ai.txt (2023) {{SPAWNING-AITXT}}: : An earlier file at "/ai.txt" published by Spawning Inc. for text-and-data-mining opt-out, scoped narrowly to TDM permission per file pattern. The format defined in this document is a strict superset, covering training, scraping, indexing, caching, per-agent rules, licensing, and attribution. The present document acknowledges Spawning's prior use of the name and positions itself as a successor declaration surface rather than a competing one." "The "ai.txt" file MUST be served at: ~~~ https://example.com/.well-known/ai.txt ~~~" "The file MUST be served with Content-Type "text/plain; charset=utf-8"." "These fields declare site-wide defaults. Each accepts "allow" or "deny". The value "conditional" is valid only for the Training field, where it activates the per-path rules defined in the Training Path Fields section; implementations encountering "conditional" on any other field SHOULD treat it as "deny"." "Training (OPTIONAL, default "deny"):" "Scraping (OPTIONAL, default "allow"):" "Indexing (OPTIONAL, default "allow"):" "Caching (OPTIONAL, default "allow"):" "When Training is "conditional", these fields specify per-path rules: Training-Allow (OPTIONAL): : Glob pattern for paths where training is permitted. Training-Deny (OPTIONAL): : Glob pattern for paths where training is denied." "Training-License (OPTIONAL): : SPDX license identifier {{SPDX}} for AI training use (e.g., "CC-BY-4.0")." "Agent blocks declare per-agent policy overrides. The wildcard "*" sets the default for all agents." "in the JSON form, the "specVersion" member, the "policies" member (with all four of its "training", "scraping", "indexing", and "caching" members), and the "agents" member are REQUIRED." ""ai.txt" is advisory. It declares the site owner's policy. Compliance is expected in good faith but is not enforced by the file itself." (https://github.com/kaylacar/ai-txt/blob/master/ietf/draft-car-ai-txt-wellknown-00.md) [^ref-6]: Kayla Cardillo, GitHub - kaylacar/ai-txt: ai.txt — open standard proposal for declaring a website's AI policy (created 2026-02-22 UTC, pushed 2026-07-18 UTC), checked 2026-10-30: "IETF Internet-Draft: draft-car-ai-txt-wellknown-00 — published on the Datatracker as an individual submission (under review, not adopted; expires 2026-12-14). Local source: ietf/draft-car-ai-txt-wellknown-00.md." "IANA Well-Known URI registrations: ai.txt (#76) and ai.json (#77) — filed, under review." "npx @ai-txt/core check https://example.com --agent ClaudeBot" (https://github.com/kaylacar/ai-txt) [^ref-7]: Kayla Cardillo, ai-txt/SPEC.md at master · kaylacar/ai-txt (ai.txt Specification, Version 1.0), checked 2026-10-30: "ai.txt Specification" "Version: 1.0" (https://github.com/kaylacar/ai-txt/blob/master/SPEC.md) [^ref-8]: Mark Nottingham (@mnot, IANA Designated Expert) and Kayla Cardillo (@kaylacar), Registration request: ai.txt · Issue #76 · protocol-registries/well-known-uris (opened 2026-02-22 UTC, closed 2026-03-16 UTC), checked 2026-10-30: "The name you have chosen is very generic, but the specification you refer to does not have broad community discussion or implementation. Furthermore, the specification is in a private GitHub repository, which is not suitable for registration due to its lack of stability." "Accordingly, it's not suitable for registration at this time." "2026-03-16T04:27:22Z" (https://github.com/protocol-registries/well-known-uris/issues/76) [^ref-9]: Mark Nottingham (@mnot, IANA Designated Expert), Registration request: ai.json · Issue #77 · protocol-registries/well-known-uris (opened 2026-02-22 UTC, closed 2026-03-16 UTC), checked 2026-10-30: "Accordingly, it's not suitable for registration at this time. I would suggest you either: 1. Re-submit with a more suitably specific name (e.g., kaylacar-ai.json) and publish a specification in a more durable way (e.g., as an Independent Stream RFC, through W3C or similar processes), or 2. Propose your specification for adoption by the IETF" "2026-03-16T04:27:41Z" (https://github.com/protocol-registries/well-known-uris/issues/77) [^ref-10]: IANA, Well-Known URIs registry CSV (neither ai.txt nor ai.json is registered), checked 2026-10-30: "URI Suffix,Reference,Status,Change Controller,Related Information,Date Registered,Date Modified" (https://www.iana.org/assignments/well-known-uris/well-known-uris-1.csv) [^ref-11]: menro, GitHub - menro/ai.txt: Proposed Standard for AI.txt (created 2023-05-18 UTC, pushed 2023-05-19 UTC), checked 2026-10-30: "AI.txt: A Guide for Preparing Website Content for Large Language Models" "In essence, AI.txt is more akin to RSS feeds than robots.txt." "2023-05-18T04:41:11Z" (https://github.com/menro/ai.txt) [^ref-12]: Yuekang Li et al., [2505.07834] ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet, submitted 2025-05-02, checked 2026-10-30: "Title: ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet" "[Submitted on 2 May 2025]" "Furthermore, we propose two compliance mechanisms: XML-based programmatic enforcement and natural language prompt integration" (https://arxiv.org/abs/2505.07834) [^ref-13]: Yuekang Li et al., ai.txt: A Domain-Specific Language for Guiding AI Interactions with the Internet (arXiv HTML v1), 2025-05-02, checked 2026-10-30: "Disallow: Return" "Disallow: Train" (https://arxiv.org/html/2505.07834v1) [^ref-14]: aitxt.ing, ai.txt — Universal Standard for AI Context, checked 2026-10-30: "ai.txt is a universal standard for describing any resource in plain text, enabling AI agents to discover context, learn facts directly, and stop hallucinating." "Prior Art Several other files share the name or the neighbourhood. They solve different problems and coexist with this one: Spawning ai.txt (2023) — a robots.txt-style file for AI training consent. Permissions, not context. IETF draft-car-ai-txt-wellknown (2026) — an individual Internet-Draft registering /.well-known/ai.txt and /.well-known/ai.json : key/value declarations of training, scraping, indexing and licensing policy, positioned as the successor to Spawning's file." "This specification is about context and boundaries in prose. It makes no claim on the permission-signalling space." (https://aitxt.ing/) [^ref-15]: WordPress Openverse API, /ai.txt on api.openverse.org, checked 2026-10-30: "# Spawning AI # Prevent datasets from using the following file types User-Agent: * Disallow: *.txt" (https://api.openverse.org/ai.txt) [^ref-16]: Republik, /ai.txt on www.republik.ch, checked 2026-10-30: "# Spawning AI # Prevent datasets from using the following file types User-Agent: * Disallow: / Disallow: *" (https://www.republik.ch/ai.txt) [^ref-17]: MLTSHP, /ai.txt on mltshp.com, checked 2026-10-30: "# Spawning AI # Prevent datasets from using the following file types User-Agent: * Disallow: *.txt" (https://mltshp.com/ai.txt) [^ref-18]: makigas, /ai.txt on www.makigas.es, checked 2026-10-30: "# Spawning AI # Prevent datasets from using the following file types User-Agent: * Disallow: *.txt" (https://www.makigas.es/ai.txt) [^ref-19]: thayn.me, /.well-known/ai.txt, checked 2026-10-30: "# ai.txt — AI usage policy for thayn.me # IETF draft: https://datatracker.ietf.org/doc/draft-car-ai-txt-wellknown/" "Spec-Version: 1.0 Site-Name: thayn.me Site-URL: https://thayn.me" (https://thayn.me/.well-known/ai.txt) [^ref-20]: Spawning Inc., datadiligence.rules package — datadiligence 0.1.7.post1.dev1+gea09739 documentation, checked 2026-10-30: "This module wraps HTTP calls to the Spawning AI API. See Spawning API documentation here: https://opts-api.spawningaiapi.com/docs ." (https://datadiligence.readthedocs.io/en/latest/api/datadiligence.rules.html) [^ref-21]: ZenphotoCMS User Guide, Protecting your site from AI scraping, 2024-03-26, checked 2026-10-30: "There is also the Initiative Spawning that aims to create tools to provide a way to block or consent AI access by opting out. There you can add yourself to a registry and create a specific ai.txt which is similar to robots.txt: https://site.spawning.ai/spawning-ai-txt#ai-text-generator . Sadly it is more a concept of an opt-out infrastructure and so far only two AI companies seems to follow that registry." (https://www.zenphoto.org/news/protecting-your-site-from-ai-scraping/) [^ref-22]: European Commission (DG CONNECT / AI Office), Stakeholder consultation on AI and copyright compliance, last updated 2025-12-01, checked 2026-10-30: "In addition to robots.txt, the study identifies seven other technical solutions (TDM reservation protocol (TDMRep); C2PA TDM Assertions; AI.txt; Do not train registry (Spawning AI); JPEG Trust core foundation v2; TDM.ai protocol (Liccium); Open Rights Data Exchange (Valunode)." (https://digital-strategy.ec.europa.eu/en/faqs/stakeholder-consultation-ai-and-copyright-compliance) [^ref-23]: David (ContextBolt), ai.txt: The Permissions File That Is Not llms.txt, published 2026-09-18, updated 2026-10-22, checked 2026-10-30: "ai.txt is a training opt-out, not a content map. The file on real sites comes from Spawning’s 2023 generator. It borrows robots.txt syntax and applies it to file types, so a line like Disallow: *.jpg means “do not train on my images”." "Five specs share the name. Spawning’s opt-out, a 2023 GitHub proposal, a May 2025 arXiv paper, a 2026 anti-hallucination format at aitxt.ing, and a June 2026 individual IETF draft. None is a standard." "No major crawler documents reading it. OpenAI, Google, Anthropic and Perplexity all point you at robots.txt. Spawning relays the file to its own partners through its API, and that is the extent of it." (https://contextbolt.com/blog/ai-txt/)