agenticweb.wiki

AIPREF Content-Usage Attachment

Last checked · 47 primary sources

This page is about draft-ietf-aipref-attach, the IETF AI Preferences (aipref) working-group Internet-Draft that says where a usage preference is published: in a Content-Usage HTTP response header and in a Content-Usage rule in robots.txt [1]. It is not the AIPREF vocabulary (draft-ietf-aipref-vocab), which defines what labels such as train-ai and search mean [18]. It is not Cloudflare's Content Signals Policy, which uses its own Content-Signal line with the signals search, ai-input and ai-train [47]; not the W3C community group's TDMRep; and not AI crawler user-agent tokens in robots.txt, which allow or block named crawlers rather than state a use preference. Each has its own page.

Overview

The AIPREF attachment draft is the IETF working-group document that tells a publisher where to put an AI usage preference so that crawlers find it while fetching content: "Methods are defined for associating usage preferences with content that is obtained using the HTTP protocol. This document defines attachment methods using the Robots Exclusion Protocol and HTTP header fields." [1]. It adds a Content-Usage header to HTTP responses and a Content-Usage rule to robots.txt, and it would formally update the Robots Exclusion Protocol to allow that rule [1]. The meaning of the preference, such as whether AI training or search is allowed, comes from the separate vocabulary draft; this draft only carries the string [1][18]. Like the vocabulary, it states a preference rather than enforcing one, and the working group has not yet sent it to the IESG [18][2].

How it works

Who uses it

Sites that serve a Content-Usage preference, fetched without following redirects on 2026-10-29 and re-checked in the self-check. Five of the seven sites named here use the label "ai": Content-Usage: ai=n is the example in the individual draft [10], and by -02 the draft's example was Content-Usage: train-ai=n [12]. The current vocabulary has no "ai" label and says "when processing a parsed Dictionary to obtain preferences, any unknown labels MUST be ignored." [18]. Launchpad, Ubuntu Changelogs, They Said So and Gazzetta del Sud carry the comment # Source: https://robotstxt.com/ai [27][31][32][35]; that template page now shows Content-Usage: train-ai=n [36]. Also seen, beyond the five listed: Gazzetta del Sud's robots.txt has Content-Usage: ai=n [35], and dri.es sends the header content-usage: ai=y next to a content-signal header [34]. No Content-Usage line was found in the robots.txt files of www.ietf.org, www.mozilla.org, blog.cloudflare.com, en.wikipedia.org, www.nytimes.com, www.theguardian.com or www.semafor.com (checked 2026-10-29).

Who reads it

No AI crawler operator, AI vendor or search engine was found saying, with a date, that its crawlers read Content-Usage (checked 2026-10-29). Google, whose Gary Illyes co-wrote the draft, lists the robots.txt fields it supports and Content-Usage is not among them [37]. A guide says: "I could not find a primary source from a major crawler operator saying it reads Content-Usage, so treat support as unknown and uneven." (2026-10-11) [38]. Tools exist on the publishing and auditing side: Nuxt Robots writes the line ("The Content-Usage directive follows the IETF AI Preferences specification, providing a standardized way to express automation preferences.") [42], and PEAC parses it: "PEAC records these signals as evidence but does not enforce them." [43]. The Cloudflare statements below are about its own Content Signals Policy and the IETF work, and do not say it reads Content-Usage.

Recent coverage

See also

References

  1. G. Illyes (Google) and M. Thomson (Mozilla), Associating AI Usage Preferences with Content in HTTP, draft-ietf-aipref-attach-05, Internet-Draft, 19 August 2026, checked 2026-10-29: "Methods are defined for associating usage preferences with content that is obtained using the HTTP protocol. This document defines attachment methods using the Robots Exclusion Protocol and HTTP header fields." "This document updates RFC 9309 to allow for the inclusion of usage preferences." "Updates: 9309 (if approved)" "This Internet-Draft will expire on 20 February 2027." "This document only defines how the strings or byte sequences are conveyed so that statements of preference can be associated with content." "Content-Usage: train-ai=n This is some content." "The Content-Usage field is a structured field dictionary, as defined in Section 3.2 of [FIELDS]. This field follows the vocabulary and processing rules in [VOCAB]." "Informally, usage preferences apply to the content of a message, not the resource." "The Content-Usage field does not have any special effect on caching." "This creates a two-stage arrangement that distinguishes acquisition and usage. Acquisition relies on Allow/Disallow rules; usage preference relies on Content-Usage rules." "Usage preferences apply only to those resources that can be crawled according to Allow/Disallow rules; no preferences are implied for resources that are disallowed." "The ordering of rules in a group carries no semantics." "Note that this differs from the Allow/Disallow rules, where a conflict leads to the more permissive option, allowing crawling." "This means that updates to "robots.txt" do not retroactively apply to resources." "User-Agent: * Allow: / Disallow: /never/ Content-Usage: train-ai=n Content-Usage: /ai-ok/ train-ai=y" "That crawler would be able to obtain all content and apply usage preferences of "ai=y" as defined in [VOCAB]." "Field Name: Content-Usage Status: permanent Structured Type: Dictionary" "draft-ietf-aipref-vocab-07, 19 August 2026" — https://www.ietf.org/archive/id/draft-ietf-aipref-attach-05.txt
  2. IETF Datatracker, draft-ietf-aipref-attach document page, checked 2026-10-29: "Document Type Active Internet-Draft (aipref WG) Authors Gary Illyes, Martin Thomson Last updated 2026-08-18 Replaces draft-it-aipref-attachment" "Intended RFC status Proposed Standard" "WG state WG Document Associated WG milestone Aug 2026 Standard track protocol specification(s) to associate the AI-related preferences to the IESG for publication" "IESG state I-D Exists" — https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/
  3. IETF Datatracker, draft-ietf-aipref-attach history (dates as shown by the Datatracker), checked 2026-10-29: "2025-06-15 00 Mark Nottingham WG -00 approved" "2025-09-04 03 Mark Nottingham IETF WG state changed to In WG Last Call from WG Document" "2025-11-03 04 Mark Nottingham IETF WG state changed to WG Document from In WG Last Call" "2026-05-01 04 (System) Document has expired" "2026-08-18 05 Martin Thomson New version available: draft-ietf-aipref-attach-05.txt" — https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/history/
  4. IETF Datatracker API, draft-ietf-aipref-attach document record, checked 2026-10-29: ""rev": "05"" ""time": "2026-08-19T04:25:16Z"" ""expires": "2027-02-20T04:25:16Z"" — https://datatracker.ietf.org/api/v1/doc/document/draft-ietf-aipref-attach/?format=json
  5. IETF Datatracker API, new-revision events for draft-ietf-aipref-attach (posting times in UTC), checked 2026-10-29: ""total_count": 6" — https://datatracker.ietf.org/api/v1/doc/newrevisiondocevent/?doc__name=draft-ietf-aipref-attach&format=json&limit=50
  6. IETF Datatracker API, documents of type RFC in group aipref, checked 2026-10-29: ""total_count": 0" — https://datatracker.ietf.org/api/v1/doc/document/?group__acronym=aipref&type=rfc&format=json
  7. IETF Datatracker, AI Preferences (aipref) working group, About page with charter and milestones, checked 2026-10-29: "Chairs Mark Nottingham, Suresh Krishnan Area Director Mike Bishop" "Standard track document(s) describing means of attaching or associating those preferences with content in IETF-defined protocols and formats, including but not limited to using Well-Known URIs (RFC 8615) such as the Robots Exclusion Protocol (RFC 9309), and HTTP response header fields" "2026-08-31 Aug 2026 Standard track protocol specification(s) to associate the AI-related preferences to the IESG for publication draft-ietf-aipref-attach" — https://datatracker.ietf.org/wg/aipref/about/
  8. IETF Datatracker, aipref working group history, checked 2026-10-29: "2025-09-23 Mike Bishop Changed milestone "Standard track protocol specification(s) to associate the AI-related preferences to the IESG for publication", set due date to August 2026 from August 2025" — https://datatracker.ietf.org/wg/aipref/history/
  9. IETF Datatracker, draft-it-aipref-attachment, checked 2026-10-29: "Replaced Internet-Draft (candidate for aipref WG) Expired & archived Authors Gary Illyes, Martin Thomson Last updated 2025-05-27 Replaced by draft-ietf-aipref-attach" — https://datatracker.ietf.org/doc/draft-it-aipref-attachment/
  10. G. Illyes and M. Thomson, draft-it-aipref-attachment-00, individual Internet-Draft, 28 May 2025, checked 2026-10-29: "Indicating Preferences Regarding Content Usage" "Content-Usage: ai=n" — https://www.ietf.org/archive/id/draft-it-aipref-attachment-00.txt
  11. Mark Nottingham, [ai-control] Call for Adoption: draft-it-aipref-attachment-00, ai-control mailing list, 2025-05-28 (+1000), checked 2026-10-29: "As you might recall, in Brussels we asked for Martin and Gary to draft a proposed document for adoption as our attachment work item." "As before, adopting a document does not imply consensus on its contents; issues can and will be raised to change it." — https://mailarchive.ietf.org/arch/msg/ai-control/MTCUmLJTNzLmLboXuJ1i2uSZ-XI/
  12. G. Illyes and M. Thomson, draft-ietf-aipref-attach-02, 21 July 2025, checked 2026-10-29: "Associating AI Usage Preferences With Content" "Content-Usage: train-ai=n" — https://www.ietf.org/archive/id/draft-ietf-aipref-attach-02.txt
  13. G. Illyes and M. Thomson, draft-ietf-aipref-attach-03, 5 September 2025, checked 2026-10-29: "Associating AI Usage Preferences with Content in HTTP draft-ietf-aipref-attach-03" — https://www.ietf.org/archive/id/draft-ietf-aipref-attach-03.txt
  14. IETF Datatracker for the aipref chairs, [ai-control] WG Last Call: draft-ietf-aipref-attach-03 (Ends 2025-09-18), ai-control mailing list, 2025-09-04 (-0700), checked 2026-10-29: "This message starts a 2-week WG Last Call for this document." — https://mailarchive.ietf.org/arch/msg/ai-control/LF4qTxITge4ID8QPsjOcNUUrDO4/
  15. Mark Nottingham and Suresh Krishnan, IETF Blog, Enabling Publishers to Express Preferences for AI Crawlers: An Update on the AIPREF Working Group, 2025-10-30, checked 2026-10-29: "The response was not one we anticipated: several participants raised fundamental issues about the nature of the Internet-Drafts (I-Ds), forcing the group to reconsider its approach." — https://www.ietf.org/blog/ai-pref-update/
  16. G. Illyes and M. Thomson, draft-ietf-aipref-attach-04, 28 October 2025, checked 2026-10-29: "Expires: 1 May 2026" — https://www.ietf.org/archive/id/draft-ietf-aipref-attach-04.txt
  17. Martin Thomson, [ai-control] New drafts for the 2026-08 meeting, ai-control mailing list, 2026-08-19, checked 2026-10-29: "There are more changes to the attachment this time, which are mostly to address drift in the vocabulary document." — https://mailarchive.ietf.org/arch/msg/ai-control/NfkU6egM1OEU1HNyKf809D5xDxU/
  18. P. Keller (Open Future) and M. Thomson (Mozilla, editor), A Vocabulary For Expressing AI Usage Preferences, draft-ietf-aipref-vocab-08, 14 September 2026, checked 2026-10-29: "when processing a parsed Dictionary to obtain preferences, any unknown labels MUST be ignored." "If any statement of preference indicates that the usage is disallowed, the result is that the usage is disallowed." "An entity that receives usage preferences has a choice whether to follow those preferences." "Preferences are not a security mechanism." — https://www.ietf.org/archive/id/draft-ietf-aipref-vocab-08.txt
  19. Tyler Martin, [ai-control] Re: [Question] Priority conflict between HTTP Header and Robots.txt in draft-ietf-aipref-attach, ai-control mailing list, 2026-03-02, checked 2026-10-29: "Precedence should follow signaling specificity, not attachment mechanism." — https://mailarchive.ietf.org/arch/msg/ai-control/f4m54TzYVGLah-_cGmfe1n6tUS0/
  20. GitHub API, ietf-wg-aipref/drafts issue #103, "Should the path in Content-Usage of Robots.txt be part of a structured field?" (opened by lrosenthol, 2025-07-21 UTC, still open), checked 2026-10-29: "I would suggest that the entire value of `Content-Usage` be a proper structured field value enabling a single parser." — https://api.github.com/repos/ietf-wg-aipref/drafts/issues/103
  21. IETF Datatracker, AI Preferences minutes, IETF 125 (2026-03-16), checked 2026-10-29: "What we're running up against is because we're chartered in a way where the vocabulary is separate from the attachment mechanism." — https://datatracker.ietf.org/doc/minutes-125-aipref/
  22. IETF Datatracker, AI Preferences interim meeting minutes, 14–16 April 2026, checked 2026-10-29: "June 2026: Virtual interim meeting focusing on "use" category proposals and restarting discussions on the attachment specification." — https://datatracker.ietf.org/doc/minutes-interim-2026-aipref-01-202604141315/
  23. IANA, Hypertext Transfer Protocol (HTTP) Field Name Registry (text), checked 2026-10-29; no Content-Usage entry: "Hypertext Transfer Protocol (HTTP) Field Name Registry" — https://www.iana.org/assignments/http-fields/http-fields.txt
  24. Editor's copy of the attachment draft on GitHub Pages, checked 2026-10-29: "Associating AI Usage Preferences with Content in HTTP" — https://ietf-wg-aipref.github.io/drafts/draft-ietf-aipref-attach.html
  25. SpinXO, robots.txt, checked 2026-10-29: "AI usage preference (draft-ietf-aipref-attach)." "Content-Usage: train-ai=y, search=y" — https://www.spinxo.com/robots.txt
  26. SpinXO, home page HTTP response header, checked 2026-10-29: "content-usage: train-ai=y, search=y" — https://www.spinxo.com/
  27. Launchpad, robots.txt, checked 2026-10-29: "Content-Usage: ai=n" — https://launchpad.net/robots.txt
  28. Launchpad Bugs, robots.txt, checked 2026-10-29: "Content-Usage: ai=n" — https://bugs.launchpad.net/robots.txt
  29. Launchpad Code, robots.txt, checked 2026-10-29: "Content-Usage: ai=n" — https://code.launchpad.net/robots.txt
  30. Launchpad Answers, robots.txt, checked 2026-10-29: "Content-Usage: ai=n" — https://answers.launchpad.net/robots.txt
  31. Ubuntu Changelogs, robots.txt, checked 2026-10-29: "Content-Usage: ai=n" — https://changelogs.ubuntu.com/robots.txt
  32. They Said So, robots.txt, checked 2026-10-29: "Content-Usage: ai=n" — https://theysaidso.com/robots.txt
  33. follow.it, robots.txt, checked 2026-10-29: "# START nuxt-robots (indexable)" "Content-Usage: bots=y, train-ai=n" — https://follow.it/robots.txt
  34. Dries Buytaert, dri.es home page HTTP response headers, checked 2026-10-29: "content-usage: ai=y" "content-signal: ai-train=yes, search=yes, ai-input=yes" — https://dri.es/
  35. Gazzetta del Sud, robots.txt, checked 2026-10-29: "Content-Usage: ai=n" — https://gazzettadelsud.it/robots.txt
  36. robotstxt.com, AI robots.txt template page, checked 2026-10-29: "Content-Usage: train-ai=n" — https://robotstxt.com/ai
  37. Google for Developers, How Google Interprets the robots.txt Specification, last updated 2026-10-23, checked 2026-10-29: "Google supports the following fields (other fields such as crawl-delay aren't supported):" — https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec
  38. IMozz, ai SiteReady, Content-Usage (aipref): robots.txt and HTTP header snippets, 2026-10-11, checked 2026-10-29: "I could not find a primary source from a major crawler operator saying it reads Content-Usage, so treat support as unknown and uneven." — https://www.ai-siteready.com/blog/content-usage-aipref-robots-txt-http
  39. Michael Tremante, [ai-control] Cloudflare's Content Signal Policy, ai-control mailing list, 2025-09-24, checked 2026-10-29: "We see this as an interim measure to address immediate needs, while work is done towards standardization." — https://mailarchive.ietf.org/arch/msg/ai-control/EANb9b9bljOanLBFm5FRerYlxyI/
  40. Bryan Becker, Cloudflare Blog, Have it both ways: stay discoverable in search while disallowing AI training, 2026-09-15, checked 2026-10-29: "As standards such as ai-prefs mature, we will revisit this approach." — https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/
  41. Cloudflare press release, Cloudflare Helps End the Search-or-AI-Training Tradeoff, 2026-09-15, checked 2026-10-29: "Today's controls are designed to work alongside these emerging standards." — https://www.cloudflare.com/press/press-releases/2026/cloudflare-helps-end-the-search-or-ai-training-tradeoff/
  42. Nuxt SEO, AI Directives · Nuxt Robots, last updated 2026-07-16, checked 2026-10-29: "The Content-Usage directive follows the IETF AI Preferences specification, providing a standardized way to express automation preferences." — https://nuxtseo.com/docs/robots/guides/ai-directives
  43. PEAC Protocol, Content Signals integration page (undated), checked 2026-10-29: "PEAC records these signals as evidence but does not enforce them." — https://www.peacprotocol.org/integrations/content-signals
  44. LLM4Agents, AIPREF at the deadline: auditing the vocabulary agents are supposed to read, 2026-08-29, checked 2026-10-29: "The three Content-Usage deployments are worth naming, because there are only three. launchpad.net and semafor.com both emit Content-Usage: ai=n." — https://llm4agents.com/blog/aipref-vocabulary-agent-access-audit
  45. Media Morpheus Research, Can You Tell an AI Not to Train on Your Site? The IETF Is Standardising How to Ask., 2026-09-26, checked 2026-10-29: "Neither has been submitted to the IESG." — https://mediamorpheuspress.com/blog/can-you-tell-an-ai-not-to-train-on-your-site
  46. AI Crawler Check, Content-Usage in robots.txt: The IETF AIPREF Standard Explained, 2026-10-18, checked 2026-10-29: "No major AI company has said publicly that its crawler reads Content-Usage yet, while every major one documents its tokens." — https://aicrawlercheck.com/blog/ietf-aipref-content-usage-robots-txt
  47. Will Allen, Cloudflare Blog, Giving users choice with Cloudflare's new Content Signals Policy, 2025-09-24, checked 2026-10-29: "This policy defines three content signals - search, ai-input, and ai-train - and their relevance to crawlers." — https://blog.cloudflare.com/content-signals-policy/

JSON · Markdown · OKF v0.2