agenticweb.wiki

Content-Signal directive

Permit Content-Signal is a machine-readable policy directive introduced by Cloudflare's Content Signals Policy that publishers place in robots.txt or HTTP response headers (Content-Signal: search=yes, ai-input=yes, ai-train=no) to express separate permissions for search indexing (search), live RAG/grounding retrieval (ai-input), and foundation model training (ai-train). It complements per-bot User-agent rules (GPTBot, ClaudeBot, OAI-SearchBot) and aligns with the IETF AIPREF vocabulary effort. Used by: cloudflare-content-signals-policy, ai-crawler-tokens-robots.

Instances

Not applicable.

See also

References

  1. https://blog.cloudflare.com/content-signals-policy/ (2026-09-05) VERIFIED
  2. https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/ (2026-09-05) VERIFIED

JSON · Markdown · OKF v0.2