--- type: Agentic Web Standard title: "Content Signals Policy" description: "May content fetched under a robots.txt allow rule also be used for search indexing, as live input to an AI answer, or for AI model training?" resource: "https://contentsignals.org/" tags: ["permit", "document-format", "single-vendor", "control-ai-crawling"] status: stable generated: { by: agenticweb-builder/1.0, at: 2026-10-30T00:00:00Z } verified: { by: process:agenticweb-engine, at: 2026-10-29T00:00:00Z } stale_after: 2026-11-28T00:00:00Z sources: - id: ref-1 resource: "https://blog.cloudflare.com/content-signals-policy/" title: "Will Allen, Cloudflare Blog, Giving users choice with Cloudflare's new Content Signals Policy, 2025-09-24, checked 2026-10-29: \"September 24, 2025 Giving users choice with Cloudflare's new Content Signals Policy Will Allen\" \"we are launching the Content Signals Policy.\" \"This policy is a new addition to robots.txt that allows you to express your preferences for how your content can be used after it has been accessed.\" \"This policy defines three content signals - search, ai-input, and ai-train - and their relevance to crawlers.\" \"A website operator can then announce their specific preferences in machine-readable text using comma-delimited, 'yes' or 'no' syntax.\" \"We will not serve an \"ai-input\" signal for our managed robots.txt customers.\" \"Starting today, we also will serve the commented, human-readable Content Signals Policy for any free customer zone that does not have an existing robots.txt file.\" \"Customers with an existing robots.txt file will see no change.\" \"we are releasing this policy under a CC0 License, which allows anyone to implement and use it freely.\" \"It's important to remember that content signals express preferences; they are not technical countermeasures against scraping.\"" last_modified: 2025-09-24T00:00:00Z - id: ref-2 resource: "https://www.cloudflare.com/press/press-releases/2025/cloudflare-gives-creators-new-tool-to-control-use-of-their-content/" title: "Cloudflare press release, Cloudflare Gives Creators New Tool to Control Use of Their Content, 2025-09-24, checked 2026-10-29: \"September 24, 2025 Cloudflare Gives Creators New Tool to Control Use of Their Content\" \"today introduced its latest way to help website owners and publishers gain more control over their content.\"" last_modified: 2025-09-24T00:00:00Z - id: ref-3 resource: "https://contentsignals.org/" title: "Cloudflare, Content Signals (contentsignals.org; the site is a single-page app, so its text was checked in the JavaScript bundle it loads, https://contentsignals.org/assets/index-DmI7GCMY.js), checked 2026-10-29: \"Introducing Content Signals, Cloudflare's implementation of a mechanism for allowing website publishers to declare how automated systems should use their content.\" \"(a) If a content-signal = yes, you may collect content for the corresponding use.\" \"(b) If a content-signal = no, you may not collect content for the corresponding use.\" \"(c) If the website operator does not include a content signal for a corresponding use, the website operator neither grants nor restricts permission via content signal with respect to the corresponding use.\" \"search: building a search index and providing search results (e.g., returning hyperlinks and short excerpts from your website's contents). Search does not include providing AI-generated search summaries.\" \"ai-input: inputting content into one or more AI models (e.g., retrieval augmented generation, grounding, or other real-time taking of content for generative AI search answers).\" \"ai-train: training or fine-tuning AI models.\" \"ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.\"" last_modified: 2026-10-29T00:00:00Z - id: ref-4 resource: "https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/" title: "Cloudflare developer documentation, robots.txt setting (Cloudflare bot solutions docs), checked 2026-10-29: \"Last updated Oct 22, 2026\" \"Bot Preference Sync generates managed robots.txt directives from your Search, Agent, and Training policies when at least one policy is set to Block, Block on pages with ads, or, for Training, Disallow.\" \"Cloudflare prepends managed directives and the Content Signals Policy to your robots.txt only when Enable Bot Preference Sync is turned on and at least one of your Search, Agent, or Training policies is set to Block, Block on pages with ads, or, for Training, Disallow.\" \"With every policy set to allow, there is nothing to prepend and your robots.txt is served unchanged.\" \"Domains on the Free plan that do not have their own robots.txt file and do not use the managed robots.txt feature will display the Content Signals Policy when a crawler requests the robots.txt file for your domain.\" \"The Content Signals Policy defines these categories but does not express any specific preferences about your content.\" \"It adds a fourth field alongside the existing search, ai-input, and ai-train signals to describe what a crawler may keep and reuse after accessing your content.\" \"For customers who have turned on the managed robots.txt setting, Cloudflare adds use=reference to the managed content, in line with the existing default of search=yes,ai-train=no\" \"Content-signal: search=yes, ai-train=no, use=reference\" \"use=immediate Interact, but store and reuse nothing. use=reference Index, excerpt, and link back. use=full Summarize and reproduce.\" \"Managed robots.txt for AI crawlers is available on all plans.\" \"Google Search Console may occasionally report Syntax not understood for Content Signals and newer directives in the robots.txt standard.\"" last_modified: 2026-10-22T00:00:00Z - id: ref-5 resource: "https://contentsignals.org/robots.txt" title: "contentsignals.org/robots.txt (robots.txt fetched live checked 2026-10-29): \"Content-Signal: ai-train=yes, search=yes, ai-input=yes\"" last_modified: 2026-10-29T00:00:00Z - id: ref-6 resource: "https://www.cloudflare.com/robots.txt" title: "www.cloudflare.com/robots.txt (robots.txt fetched live checked 2026-10-29): \"Content-Signal: ai-train=yes, search=yes, ai-input=yes\"" last_modified: 2026-10-29T00:00:00Z - id: ref-7 resource: "https://mailarchive.ietf.org/arch/msg/ai-control/EANb9b9bljOanLBFm5FRerYlxyI/" title: "Michael Tremante, [ai-control] Cloudflare's Content Signal Policy, ai-control mailing list (IETF), 2025-09-24, checked 2026-10-29: \"We see this as an interim measure to address immediate needs, while work is done towards standardization.\" \"We will continue to engage in this working group and hope we can achieve a consensus position and will revisit our policy once that has happened.\"" last_modified: 2025-09-24T00:00:00Z - id: ref-8 resource: "https://datatracker.ietf.org/doc/draft-romm-aipref-contentsignals/" title: "IETF Datatracker, draft-romm-aipref-contentsignals, Vocabulary For Expressing Content Signals, checked 2026-10-29: \"Expired Internet-Draft (individual)\" \"Authors Michael Tremante, Leah Romm Last updated 2026-04-04 (Latest revision 2025-10-01)\" \"The proposal is for these categories to nest within the larger category of Automated Processing, currently envisaged in the [AIPREF-VOCAB].\"" last_modified: 2026-04-04T00:00:00Z - id: ref-9 resource: "https://blog.cloudflare.com/content-independence-day-ai-options/" title: "Jin-Hee Lee and Bryan Becker, Cloudflare Blog, Your site, your rules: new AI traffic options for all customers, 2026-07-01, checked 2026-10-29: \"July 1, 2026 Your site, your rules: new AI traffic options for all customers Jin-Hee Lee and Bryan Becker\" \"we're testing a new signal, use, that extends Content Signals and lives in your robots.txt.\" \"As with all other items listed in the robots.txt file, the values of content use signal a website owner's preference, rather than issuing blocks directly.\"" last_modified: 2026-07-01T00:00:00Z - id: ref-10 resource: "https://blog.cloudflare.com/bot-preference-sync/" title: "Jin-Hee Lee, Cloudflare Blog, Say it once: introducing Bot Preference Sync, 2026-08-21, checked 2026-10-29: \"August 21, 2026 Say it once: introducing Bot Preference Sync Jin-Hee Lee\"" last_modified: 2026-08-21T00:00:00Z - id: ref-11 resource: "https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/" title: "Bryan Becker, Cloudflare Blog, Have it both ways: stay discoverable in search while disallowing AI training, 2026-09-15, checked 2026-10-29: \"September 15, 2026 Have it both ways: stay discoverable in search while disallowing AI training Bryan Becker\" \"Managed Robots.txt will be deprecated in favor of Bot Preference Sync. Customers who enabled Managed Robots.txt will migrate to the new system.\" \"As standards such as ai-prefs mature, we will revisit this approach.\"" last_modified: 2026-09-15T00:00:00Z - id: ref-12 resource: "https://www.cloudflare.com/press/press-releases/2026/cloudflare-helps-end-the-search-or-ai-training-tradeoff/" title: "Cloudflare press release, Cloudflare Helps End the Search-or-AI-Training Tradeoff, 2026-09-15, checked 2026-10-29: \"Today's controls are designed to work alongside these emerging standards.\"" last_modified: 2026-09-15T00:00:00Z - id: ref-13 resource: "https://developers.cloudflare.com/changelog/post/2026-08-31-crawl-content-use/" title: "Cloudflare changelog, Crawl endpoint now respects the Content Signals use directive, 2026-08-31, checked 2026-10-29: \"The /crawl endpoint now respects the use directive of the Content Signals ↗︎ standard, letting site owners express the maximum level at which their content may be used.\"" last_modified: 2026-08-31T00:00:00Z - id: ref-14 resource: "https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/" title: "Cloudflare Browser Run documentation, /crawl - Crawl web content, checked 2026-10-29: \"The /crawl endpoint enforces both the yes/no purpose directives (search, ai-input, ai-train) and the use level directive.\" \"Content Signals are trust-based.\"" last_modified: 2026-10-29T00:00:00Z - id: ref-15 resource: "https://developers.cloudflare.com/changelog/post/2025-10-21-track-robots-txt/" title: "Cloudflare changelog, New Robots.txt tab for tracking crawler compliance, 2025-10-21, checked 2026-10-29: \"Check whether your robots.txt files contain Content Signals ↗︎ directives for AI training, search, and AI input.\"" last_modified: 2025-10-21T00:00:00Z - id: ref-16 resource: "https://developers.cloudflare.com/changelog/post/2026-07-13-markdown-for-agents-header-preservation/" title: "Cloudflare changelog, Origin Content Signals for Markdown for Agents, 2026-07-13, checked 2026-10-29: \"If your origin sets a content-signal header, Markdown for Agents preserves it. When the origin does not send one, Cloudflare adds the default Content-Signal: ai-train=yes, search=yes, ai-input=yes.\"" last_modified: 2026-07-13T00:00:00Z - id: ref-17 resource: "https://rslstandard.org/rsl" title: "RSL Collective, Really Simple Licensing (RSL) 1.0 Specification, checked 2026-10-29: \"This category includes the Cloudflare Content Signals vocabulary and MAY include terms from additional standardized vocabularies as they become available (e.g., IETF AI Preferences).\"" last_modified: 2026-10-29T00:00:00Z - id: ref-18 resource: "https://www.seroundtable.com/google-cloudflare-content-signals-41631.html" title: "Barry Schwartz, Search Engine Roundtable, Google Says Cloudflare Content Signals Robots.txt Directive Has No Effects Whatsoever, 2026-07-06, checked 2026-10-29: \"John Mueller from Google said that the new Content Signals robots.txt directive invented by Cloudflare last year has \"no effects whatsoever for any crawler or LLM.\"\" \"AFAIK none of the crawlers / llms use the \"content-signal\" robots.txt directives. It was made up by a CDN, afaik it has no effects whatsoever for any crawler or llm.\" \"Using it just adds bloat & future maintenance to your robots.txt file.\"" last_modified: 2026-07-06T00:00:00Z - id: ref-19 resource: "https://www.searchenginejournal.com/google-answers-question-about-llms-author-txt-for-seo/581547/" title: "Roger Montti, Search Engine Journal, Google Answers Question About LLMs-Author.txt For SEO, 2026-07-06, checked 2026-10-29: \"Roger Montti July 6, 2026\" \"Cloudflare had proposed a Content-Signal directive for robots.txt.\" \"Cloudflare later used the same Content-Signal syntax as an HTTP response header automatically generated in Markdown for Agents.\" \"AFAIK none of the crawlers / llms use the \"content-signal\" robots.txt directives. It was made up by a CDN, afaik it has no effects whatsoever for any crawler or llm.\"" last_modified: 2026-07-06T00:00:00Z - id: ref-20 resource: "https://launchpad.net/robots.txt" title: "launchpad.net/robots.txt (robots.txt fetched live checked 2026-10-29): \"Content-Signal: ai-train=no, search=yes, ai-input=no\" \"Content-Usage: ai=n\"" last_modified: 2026-10-29T00:00:00Z - id: ref-21 resource: "https://docs.nvidia.com/robots.txt" title: "docs.nvidia.com/robots.txt (robots.txt fetched live checked 2026-10-29): \"Content-Signal: ai-train=yes, search=yes, ai-input=yes\"" last_modified: 2026-10-29T00:00:00Z - id: ref-22 resource: "https://render.com/robots.txt" title: "render.com/robots.txt (robots.txt fetched live checked 2026-10-29): \"Content-Signal: ai-train=yes, search=yes, ai-input=yes\"" last_modified: 2026-10-29T00:00:00Z - id: ref-23 resource: "https://linktr.ee/robots.txt" title: "linktr.ee/robots.txt (robots.txt fetched live checked 2026-10-29): \"User-agent: GPTBot Content-Signal: ai-train=no, search=yes\" \"(a) If a content-signal = yes, you may collect content for the corresponding use.\"" last_modified: 2026-10-29T00:00:00Z - id: ref-24 resource: "https://workable.com/robots.txt" title: "workable.com/robots.txt (robots.txt fetched live checked 2026-10-29): \"Content-Signal: search=yes, ai-input=yes, ai-train=no\"" last_modified: 2026-10-29T00:00:00Z --- # Content Signals Policy Content Signals Policy is an agentic-web standard (document format — Content-Signal) at step 3, Permit, governed by Cloudflare; its status is "Live since 2025-09-24". It answers: "May content fetched under a robots.txt allow rule also be used for search indexing, as live input to an AI answer, or for AI model training?" Replacing or extending robots.txt policy extension, implementation status is multi-implementation, last verified 2026-10-29. # Overview The Content Signals Policy is Cloudflare's way for a website to say, inside its robots.txt file, what a crawler may do with content after fetching it [^ref-1]. A short commented policy text defines three uses: search (a search index with links and short excerpts, not AI summaries), ai-input (feeding content to an AI model at answer time, such as retrieval augmented generation) and ai-train (training or fine-tuning) [^ref-3]. A machine-readable Content-Signal line then sets each use to yes or no, and a use left out is neither granted nor restricted [^ref-1][^ref-3]. Cloudflare released the policy under CC0, serves it in the robots.txt files it manages for customers, and since July 2026 has been testing an optional fourth field, use, that caps how much of the content a crawler may keep [^ref-1][^ref-4][^ref-9]. Why it matters for AI agents: robots.txt has said only whether a crawler may fetch a page, not what it may do with the page afterwards, and the same crawler can feed a search index, an AI answer and a training set [^ref-1][^ref-11]. Content Signals give a site a way to separate those uses in one line, and the policy text frames any restriction as a reservation of rights under Article 4 of the EU copyright directive [^ref-3]. The signals are preferences, not technical blocks [^ref-1]. Cloudflare's own /crawl service enforces them [^ref-13][^ref-14], while Google's John Mueller said in July 2026 that, as far as he knew, no crawler or LLM uses them [^ref-18]. # Examples ```http GET {origin}/robots.txt # parse: Look for 'content-signal: search=', 'content-signal: ai-input=', or 'content-signal: ai-train=' lines (as machine-readable comments) inside any robots.txt group # on absent: Website operator has not stated a content-signal preference for that use; the spec is explicit that omission means the operator neither grants nor restricts permission for that use ``` # Knowledge Graph * **Lifecycle Step**: [3 Permit](/category/permit/index.md) * **Builds on**: [AI crawler tokens in robots.txt](/okf/protocols/ai-crawler-tokens-robots.md) * **Pairs with**: [AI crawler tokens in robots.txt](/okf/protocols/ai-crawler-tokens-robots.md), [TDMRep (TDM Reservation Protocol)](/okf/protocols/tdmrep.md), [AIPREF Vocabulary (A Vocabulary For Expressing AI Usage Preferences)](/okf/protocols/aipref-vocabulary.md), [AIPREF Content-Usage Attachment](/okf/protocols/aipref-content-usage-attachment.md), [Cloudflare Pay Per Crawl](/okf/protocols/cloudflare-pay-per-crawl.md), [Markdown for Agents](/okf/protocols/cloudflare-markdown-for-agents.md) * **Related terms**: [AI crawler tokens](/okf/terms/ai-crawler-tokens.md), [Content-Signal directive](/okf/terms/content-signal.md) * **Compared in**: [robots.txt (Allow/Disallow) vs Content-Signal, AIPREF & TDMRep (classic vs agentic)](/compare/robots-txt-vs-content-signal/index.md), [Content Signals Policy vs Web Bot Auth](/compare/cloudflare-content-signals-policy-vs-webbotauth/index.md), [Content Signals Policy vs AIPREF Vocabulary (A Vocabulary For Expressing AI Usage Preferences)](/compare/cloudflare-content-signals-policy-vs-aipref-vocabulary/index.md), [AI crawler tokens in robots.txt vs Content Signals Policy](/compare/ai-crawler-tokens-robots-vs-cloudflare-content-signals-policy/index.md), [Really Simple Licensing vs Content Signals Policy](/compare/rsl-vs-cloudflare-content-signals-policy/index.md) [^ref-1]: Will Allen, Cloudflare Blog, Giving users choice with Cloudflare's new Content Signals Policy, 2025-09-24, checked 2026-10-29: "September 24, 2025 Giving users choice with Cloudflare's new Content Signals Policy Will Allen" "we are launching the Content Signals Policy." "This policy is a new addition to robots.txt that allows you to express your preferences for how your content can be used after it has been accessed." "This policy defines three content signals - search, ai-input, and ai-train - and their relevance to crawlers." "A website operator can then announce their specific preferences in machine-readable text using comma-delimited, 'yes' or 'no' syntax." "We will not serve an "ai-input" signal for our managed robots.txt customers." "Starting today, we also will serve the commented, human-readable Content Signals Policy for any free customer zone that does not have an existing robots.txt file." "Customers with an existing robots.txt file will see no change." "we are releasing this policy under a CC0 License, which allows anyone to implement and use it freely." "It's important to remember that content signals express preferences; they are not technical countermeasures against scraping." (https://blog.cloudflare.com/content-signals-policy/) [^ref-2]: Cloudflare press release, Cloudflare Gives Creators New Tool to Control Use of Their Content, 2025-09-24, checked 2026-10-29: "September 24, 2025 Cloudflare Gives Creators New Tool to Control Use of Their Content" "today introduced its latest way to help website owners and publishers gain more control over their content." (https://www.cloudflare.com/press/press-releases/2025/cloudflare-gives-creators-new-tool-to-control-use-of-their-content/) [^ref-3]: Cloudflare, Content Signals (contentsignals.org; the site is a single-page app, so its text was checked in the JavaScript bundle it loads, https://contentsignals.org/assets/index-DmI7GCMY.js), checked 2026-10-29: "Introducing Content Signals, Cloudflare's implementation of a mechanism for allowing website publishers to declare how automated systems should use their content." "(a) If a content-signal = yes, you may collect content for the corresponding use." "(b) If a content-signal = no, you may not collect content for the corresponding use." "(c) If the website operator does not include a content signal for a corresponding use, the website operator neither grants nor restricts permission via content signal with respect to the corresponding use." "search: building a search index and providing search results (e.g., returning hyperlinks and short excerpts from your website's contents). Search does not include providing AI-generated search summaries." "ai-input: inputting content into one or more AI models (e.g., retrieval augmented generation, grounding, or other real-time taking of content for generative AI search answers)." "ai-train: training or fine-tuning AI models." "ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET." (https://contentsignals.org/) [^ref-4]: Cloudflare developer documentation, robots.txt setting (Cloudflare bot solutions docs), checked 2026-10-29: "Last updated Oct 22, 2026" "Bot Preference Sync generates managed robots.txt directives from your Search, Agent, and Training policies when at least one policy is set to Block, Block on pages with ads, or, for Training, Disallow." "Cloudflare prepends managed directives and the Content Signals Policy to your robots.txt only when Enable Bot Preference Sync is turned on and at least one of your Search, Agent, or Training policies is set to Block, Block on pages with ads, or, for Training, Disallow." "With every policy set to allow, there is nothing to prepend and your robots.txt is served unchanged." "Domains on the Free plan that do not have their own robots.txt file and do not use the managed robots.txt feature will display the Content Signals Policy when a crawler requests the robots.txt file for your domain." "The Content Signals Policy defines these categories but does not express any specific preferences about your content." "It adds a fourth field alongside the existing search, ai-input, and ai-train signals to describe what a crawler may keep and reuse after accessing your content." "For customers who have turned on the managed robots.txt setting, Cloudflare adds use=reference to the managed content, in line with the existing default of search=yes,ai-train=no" "Content-signal: search=yes, ai-train=no, use=reference" "use=immediate Interact, but store and reuse nothing. use=reference Index, excerpt, and link back. use=full Summarize and reproduce." "Managed robots.txt for AI crawlers is available on all plans." "Google Search Console may occasionally report Syntax not understood for Content Signals and newer directives in the robots.txt standard." (https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) [^ref-5]: contentsignals.org/robots.txt (robots.txt fetched live checked 2026-10-29): "Content-Signal: ai-train=yes, search=yes, ai-input=yes" (https://contentsignals.org/robots.txt) [^ref-6]: www.cloudflare.com/robots.txt (robots.txt fetched live checked 2026-10-29): "Content-Signal: ai-train=yes, search=yes, ai-input=yes" (https://www.cloudflare.com/robots.txt) [^ref-7]: Michael Tremante, [ai-control] Cloudflare's Content Signal Policy, ai-control mailing list (IETF), 2025-09-24, checked 2026-10-29: "We see this as an interim measure to address immediate needs, while work is done towards standardization." "We will continue to engage in this working group and hope we can achieve a consensus position and will revisit our policy once that has happened." (https://mailarchive.ietf.org/arch/msg/ai-control/EANb9b9bljOanLBFm5FRerYlxyI/) [^ref-8]: IETF Datatracker, draft-romm-aipref-contentsignals, Vocabulary For Expressing Content Signals, checked 2026-10-29: "Expired Internet-Draft (individual)" "Authors Michael Tremante, Leah Romm Last updated 2026-04-04 (Latest revision 2025-10-01)" "The proposal is for these categories to nest within the larger category of Automated Processing, currently envisaged in the [AIPREF-VOCAB]." (https://datatracker.ietf.org/doc/draft-romm-aipref-contentsignals/) [^ref-9]: Jin-Hee Lee and Bryan Becker, Cloudflare Blog, Your site, your rules: new AI traffic options for all customers, 2026-07-01, checked 2026-10-29: "July 1, 2026 Your site, your rules: new AI traffic options for all customers Jin-Hee Lee and Bryan Becker" "we're testing a new signal, use, that extends Content Signals and lives in your robots.txt." "As with all other items listed in the robots.txt file, the values of content use signal a website owner's preference, rather than issuing blocks directly." (https://blog.cloudflare.com/content-independence-day-ai-options/) [^ref-10]: Jin-Hee Lee, Cloudflare Blog, Say it once: introducing Bot Preference Sync, 2026-08-21, checked 2026-10-29: "August 21, 2026 Say it once: introducing Bot Preference Sync Jin-Hee Lee" (https://blog.cloudflare.com/bot-preference-sync/) [^ref-11]: Bryan Becker, Cloudflare Blog, Have it both ways: stay discoverable in search while disallowing AI training, 2026-09-15, checked 2026-10-29: "September 15, 2026 Have it both ways: stay discoverable in search while disallowing AI training Bryan Becker" "Managed Robots.txt will be deprecated in favor of Bot Preference Sync. Customers who enabled Managed Robots.txt will migrate to the new system." "As standards such as ai-prefs mature, we will revisit this approach." (https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) [^ref-12]: Cloudflare press release, Cloudflare Helps End the Search-or-AI-Training Tradeoff, 2026-09-15, checked 2026-10-29: "Today's controls are designed to work alongside these emerging standards." (https://www.cloudflare.com/press/press-releases/2026/cloudflare-helps-end-the-search-or-ai-training-tradeoff/) [^ref-13]: Cloudflare changelog, Crawl endpoint now respects the Content Signals use directive, 2026-08-31, checked 2026-10-29: "The /crawl endpoint now respects the use directive of the Content Signals ↗︎ standard, letting site owners express the maximum level at which their content may be used." (https://developers.cloudflare.com/changelog/post/2026-08-31-crawl-content-use/) [^ref-14]: Cloudflare Browser Run documentation, /crawl - Crawl web content, checked 2026-10-29: "The /crawl endpoint enforces both the yes/no purpose directives (search, ai-input, ai-train) and the use level directive." "Content Signals are trust-based." (https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/) [^ref-15]: Cloudflare changelog, New Robots.txt tab for tracking crawler compliance, 2025-10-21, checked 2026-10-29: "Check whether your robots.txt files contain Content Signals ↗︎ directives for AI training, search, and AI input." (https://developers.cloudflare.com/changelog/post/2025-10-21-track-robots-txt/) [^ref-16]: Cloudflare changelog, Origin Content Signals for Markdown for Agents, 2026-07-13, checked 2026-10-29: "If your origin sets a content-signal header, Markdown for Agents preserves it. When the origin does not send one, Cloudflare adds the default Content-Signal: ai-train=yes, search=yes, ai-input=yes." (https://developers.cloudflare.com/changelog/post/2026-07-13-markdown-for-agents-header-preservation/) [^ref-17]: RSL Collective, Really Simple Licensing (RSL) 1.0 Specification, checked 2026-10-29: "This category includes the Cloudflare Content Signals vocabulary and MAY include terms from additional standardized vocabularies as they become available (e.g., IETF AI Preferences)." (https://rslstandard.org/rsl) [^ref-18]: Barry Schwartz, Search Engine Roundtable, Google Says Cloudflare Content Signals Robots.txt Directive Has No Effects Whatsoever, 2026-07-06, checked 2026-10-29: "John Mueller from Google said that the new Content Signals robots.txt directive invented by Cloudflare last year has "no effects whatsoever for any crawler or LLM."" "AFAIK none of the crawlers / llms use the "content-signal" robots.txt directives. It was made up by a CDN, afaik it has no effects whatsoever for any crawler or llm." "Using it just adds bloat & future maintenance to your robots.txt file." (https://www.seroundtable.com/google-cloudflare-content-signals-41631.html) [^ref-19]: Roger Montti, Search Engine Journal, Google Answers Question About LLMs-Author.txt For SEO, 2026-07-06, checked 2026-10-29: "Roger Montti July 6, 2026" "Cloudflare had proposed a Content-Signal directive for robots.txt." "Cloudflare later used the same Content-Signal syntax as an HTTP response header automatically generated in Markdown for Agents." "AFAIK none of the crawlers / llms use the "content-signal" robots.txt directives. It was made up by a CDN, afaik it has no effects whatsoever for any crawler or llm." (https://www.searchenginejournal.com/google-answers-question-about-llms-author-txt-for-seo/581547/) [^ref-20]: launchpad.net/robots.txt (robots.txt fetched live checked 2026-10-29): "Content-Signal: ai-train=no, search=yes, ai-input=no" "Content-Usage: ai=n" (https://launchpad.net/robots.txt) [^ref-21]: docs.nvidia.com/robots.txt (robots.txt fetched live checked 2026-10-29): "Content-Signal: ai-train=yes, search=yes, ai-input=yes" (https://docs.nvidia.com/robots.txt) [^ref-22]: render.com/robots.txt (robots.txt fetched live checked 2026-10-29): "Content-Signal: ai-train=yes, search=yes, ai-input=yes" (https://render.com/robots.txt) [^ref-23]: linktr.ee/robots.txt (robots.txt fetched live checked 2026-10-29): "User-agent: GPTBot Content-Signal: ai-train=no, search=yes" "(a) If a content-signal = yes, you may collect content for the corresponding use." (https://linktr.ee/robots.txt) [^ref-24]: workable.com/robots.txt (robots.txt fetched live checked 2026-10-29): "Content-Signal: search=yes, ai-input=yes, ai-train=no" (https://workable.com/robots.txt)