{
 "acronym": "",
 "adoption": {
  "by": null,
  "level": "unmeasured",
  "probe_run": null
 },
 "aliases": [
  "Cloudflare Content Signals Policy"
 ],
 "analog": "robots.txt policy extension",
 "collisions": [],
 "discovery": {
  "mechanism": "Machine-readable comment lines inside robots.txt",
  "path": "/robots.txt"
 },
 "dispute": {
  "claims": [
   "Cloudflare presents Content Signals as a way for publishers to express search / ai-train / ai-input preferences, and enforces its own categories at the network edge for zones it fronts (new defaults from 2026-09-15).",
   "Google's John Mueller stated on 2026-07-06 that no crawler or LLM he is aware of reads the robots.txt directive and that Google does not, so outside Cloudflare's own edge the directive is a declared preference with no confirmed consumer [3][4]."
  ],
  "summary": "Two different things carry the same name: an edge-enforced Cloudflare feature that works because Cloudflare blocks the request, and a robots.txt directive that only works if a crawler chooses to honour it. As of July 2026 Google says it does not, and no other major crawler has documented doing so."
 },
 "evidence": [
  {
   "claim": "Cloudflare's blog post, datePublished 2025-09-24 per its own page metadata, defines three content signals (search, ai-input, ai-train) as machine-readable comment lines inside robots.txt, each independently settable to yes/no, and frames restrictions as express reservations of rights under EU Directive 2019/790 Article 4.",
   "date": "2025-09-24",
   "label": "VERIFIED",
   "url": "https://blog.cloudflare.com/content-signals-policy/"
  },
  {
   "claim": "Cloudflare's own developer documentation (distinct from the announcement blog post) reproduces the exact Content Signals Policy text and confirms it ships by default in the managed robots.txt feature, and separately states this feature was already turned on for over 3.8 million domains at launch, giving an independent, dated adoption figure from Cloudflare's own materials.",
   "date": "2025-09-24",
   "label": "VERIFIED",
   "url": "https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/"
  },
  {
   "claim": "Google's John Mueller (r/TechSEO, 2026-07-06): no crawler or LLM he knows of reads the content-signal robots.txt directive; it 'has no effects whatsoever for any crawler or llm' and 'just adds bloat'. Reported independently by Search Engine Roundtable and Search Engine Journal.",
   "date": "2026-07-06",
   "label": "VERIFIED",
   "url": "https://www.seroundtable.com/google-cloudflare-content-signals-41631.html"
  },
  {
   "claim": "Search Engine Journal's account of the same statement, noting Cloudflare also emits the Content-Signal syntax as an HTTP response header in Markdown for Agents.",
   "date": "2026-07-06",
   "label": "VERIFIED",
   "url": "https://www.searchenginejournal.com/google-answers-question-about-llms-author-txt-for-seo/581547/"
  }
 ],
 "form": "well-known-file",
 "governance": "Cloudflare, developed with publisher/platform partners named in the announcement (e.g. Condé Nast, News/Media Alliance, Reddit, Pinterest, Quora, Yahoo)",
 "group": null,
 "history": [
  {
   "date": "2026-09-05",
   "note": "seeded from Exa Agent research run"
  },
  {
   "date": "2026-09-06",
   "note": "verified against blog.cloudflare.com/content-signals-policy/"
  },
  {
   "date": "2026-09-07",
   "note": "added Google's 2026-07-06 statement that no crawler honours the directive (Search Engine Roundtable, Search Engine Journal)"
  }
 ],
 "id": "cloudflare-content-signals-policy",
 "implementation": "shipping",
 "last_verified": "2026-09-07",
 "name": "Content Signals Policy",
 "native_status": "Vendor-published policy convention layered on robots.txt; not submitted to a standards body as of this verification",
 "origin": {
  "date": "2025-09-24",
  "org": "Cloudflare"
 },
 "part_of": null,
 "parties": [
  "agent-site"
 ],
 "phase": "living",
 "probe": {
  "method": "GET",
  "on_absent": "Website operator has not stated a content-signal preference for that use; the spec is explicit that omission means the operator neither grants nor restricts permission for that use",
  "parse": "Look for 'content-signal: search=', 'content-signal: ai-input=', or 'content-signal: ai-train=' lines (as machine-readable comments) inside any robots.txt group",
  "path": "{origin}/robots.txt"
 },
 "question": "May content fetched under a robots.txt allow rule also be used for search indexing, as live input to an AI answer, or for AI model training?",
 "scope": "general",
 "spec_url": "https://blog.cloudflare.com/content-signals-policy/",
 "step": "permit",
 "summary_md": "The Content Signals Policy is a Cloudflare-authored convention, announced 2025-09-24, that extends robots.txt with a machine-readable way for a website operator to state whether content a crawler is allowed to fetch may also be used for three specific downstream purposes: `search` (building a search index and returning links/excerpts, explicitly excluding AI-generated search summaries), `ai-input` (feeding content into an AI model at request time, e.g. retrieval-augmented generation or grounding), and `ai-train` (training or fine-tuning a model) [1]. It is a permit-layer refinement on top of the existing allow/disallow crawl decision: robots.txt still governs whether a page may be fetched at all, and content signals then govern what may be done with a page the crawler was already allowed to fetch.\n\nSignals are expressed as human-readable comment lines (so they degrade gracefully for crawlers that don't parse them) of the form `content-signal: search=yes, ai-train=no`. Per the spec text, `yes` permits the corresponding use, `no` prohibits it, and omitting a signal means the operator neither grants nor restricts permission for that use — a deliberately weaker default than an explicit prohibition [1]. Cloudflare frames any expressed restriction as an express reservation of rights under Article 4 of the EU Copyright Directive (2019/790), aligning its legal grounding with TDMRep's.\n\nIt does not define enforcement, and Cloudflare's own post frames it as advisory: compliance depends on crawler operators choosing to read and honour the signal, exactly like robots.txt itself. It has not, as of this verification, been submitted to IETF, W3C, or any other standards body — it remains a single-vendor convention, distinct from and overlapping with the IETF AIPREF Content-Usage/Vocabulary drafts, which use a different attachment surface (a dedicated header/directive) and a different vocabulary.\n\nAs of this verification (2026-09-06), the announcement date of 2025-09-24 is confirmed directly from the post's own structured-data metadata, matching the prior finding [1]. Cloudflare ships the mechanism directly into its own customer-facing robots.txt management tooling, giving it a shipping implementation by definition for any site on Cloudflare's platform, though no independent count of how many zones have set a non-default signal was found in this pass — adoption is unmeasured. What is unresolved: whether AI vendors (OpenAI, Anthropic, Google, Perplexity) formally commit to honouring these specific signal names in their own crawler documentation, as opposed to their existing per-bot robots.txt tokens (see ai-crawler-tokens-robots), was not confirmed in this pass.",
 "track": "single-vendor",
 "version": {
  "date": "2025-09-24",
  "label": "Announced 2025-09-24"
 }
}