robots.txt (Allow/Disallow) vs Content-Signal, AIPREF & TDMRep
RFC 9309 Robots Exclusion Protocol (/robots.txt) is the classic web primitive built for human browsers and pre-integrated services (Instruct automated crawlers which URL paths they may or may not fetch from an origin), whereas Content Signals Policy and its step 3 (Permit) peers provide the machine-native agentic surface (Content-Signal) so autonomous agents can answer: "May content fetched under a robots.txt allow rule also be used for search indexing, as live input to an AI answer, or for AI model training?" Across the agenticweb.wiki conformance panel, Content Signals Policy is multi-implementation (116 of 574 probed origins (last checked 2026-10-23)).
How robots.txt (Allow/Disallow) compares to Content-Signal, AIPREF & TDMRep at step 3 (Permit): why the classic primitive breaks for software agents, and how sites deploy both together.
| Dimension | RFC 9309 Robots Exclusion Protocol (/robots.txt) Classic web | Content Signals Policy Agentic web |
|---|---|---|
| Primary consumer | Human driving a web browser or pre-integrated developer code | Autonomous software agent, coding harness, or AI search/RAG crawler |
| Lifecycle step | Classic web baseline for step 3 (Permit) | 3 Permit: What may the agent do here? |
| Core question answered | Instruct automated crawlers which URL paths they may or may not fetch from an origin. | May content fetched under a robots.txt allow rule also be used for search indexing, as live input to an AI answer, or for AI model training? |
| Wire format & entrypoint | /robots.txt (User-agent, Allow, Disallow) | well-known file at Content-Signal |
| Failure mode for agents | Classic Allow and Disallow rules only govern whether a bot can fetch a byte stream—they cannot express what the caller is permitted to do with the content afterwards (allowing live AI search grounding while prohibiting foundation-model training). | Solved natively: The Content Signals Policy is a Cloudflare-authored convention, announced 2025-09-24, that extends robots.txt with a machine-readable way for a website operator to state whether content a crawler is allowed to fetch may also be used for three specific downstream purposes: search (building a search index and returning links/excerpts, explicitly excluding AI-generated search summaries), ai-input (feeding content into an AI model at request time, e.g. retrieval-augmented generation or grounding), and ai-train (training or fine-tuning a model). |
| Governance & status | Established IETF / W3C / industry baseline | Cloudflare, developed with publisher/platform partners named in the announcement (e.g. Condé Nast, News/Media Alliance, Reddit, Pinterest, Quora, Yahoo) — Vendor-published policy convention layered on robots.txt; not submitted to a standards body as of this verification |
| Live adoption (/check/ census) | Universal across the classic web | multi-implementation — 116 of 574 probed origins (last checked 2026-10-23) |
| Primary specification | https://www.rfc-editor.org/rfc/rfc9309 | https://blog.cloudflare.com/content-signals-policy/ |
Why robots.txt (Allow/Disallow) breaks when the caller is an agent
Classic Allow and Disallow rules only govern whether a bot can fetch a byte stream—they cannot express what the caller is permitted to do with the content afterwards (allowing live AI search grounding while prohibiting foundation-model training).
When to use which (and how they work together)
- Keep RFC 9309 Robots Exclusion Protocol (/robots.txt): You are controlling crawl budget, blocking private paths, or setting per-bot path access for named crawlers (OAI-SearchBot vs GPTBot).
- Add Content Signals Policy: You want to declare granular downstream usage permissions—such as search=yes, ai-input=yes, ai-train=no via Content-Signal, IETF AIPREF Content-Usage headers, W3C TDMRep (/.well-known/tdmrep.json), or RSL licensing links.
All agentic web standards replacing or extending robots.txt (Allow/Disallow)
- Content Signals Policy — multi-implementation · 116/574 probed origins: May content fetched under a robots.txt allow rule also be used for search indexing, as live input to an AI answer, or for AI model training?
- AI Crawler User-Agent Tokens in robots.txt — multi-implementation · 141/601 probed origins: Which named AI crawler or agent user-agent tokens may a website allow or disallow in robots.txt, and what does each token actually control (search indexing, AI training, or live user-triggered fetches)?
- AIPREF Vocabulary — none found: What controlled set of values expresses a publisher's AI-use preferences, independent of how they are attached to content?
- AIPREF Content-Usage Attachment — none found: How does a publisher attach its AI-use preferences to content served over HTTP, so an automated system can find them?
- Text and Data Mining Reservation Protocol (TDMRep) — shipping · 5/574 probed origins: Has a rights holder reserved text-and-data-mining rights over this content, and what policy governs any permitted mining?
- Really Simple Licensing (RSL) — shipping · 3/574 probed origins: What machine-readable license, price, and authorization terms govern how AI systems and crawlers may access or reuse a publisher's content?