# Useful resources Scanners, registries, indexes and standards-body pages that tell you something checkable about the agentic web. Each entry says what it does and which articles it covers. ## Documentation - [Permit](/category/permit/) [Cloudflare AI Crawl Control bot reference](https://developers.cloudflare.com/ai-crawl-control/reference/bots/) Names the AI crawlers Cloudflare recognises, with the operator, user-agent token and stated purpose of each. Covers: [AI Crawler User-Agent Tokens in robots.txt](/protocols/ai-crawler-tokens-robots/) · Last checked 2026-09-06 - [Identify](/category/identify/) [Cloudflare Web Bot Auth documentation](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/) Documents how Cloudflare verifies HTTP message signatures from bots at the edge and how a bot operator registers keys. Covers: [Web Bot Auth HTTP Message Signatures](/protocols/webbotauth/), [HTTP Message Signatures](/protocols/http-message-signatures/) · Last checked 2026-09-06 - [Read](/category/read/) [llmstxt.org](https://llmstxt.org/) The llms.txt proposal's own site: the file format, its rationale and a directory of sites publishing one. Covers: [llms.txt](/protocols/llms-txt/) · Last checked 2026-09-06 ## Indexes - [Discover](/category/discover/) [ardvark](https://ardvark.no/) Crawls the web for ai-catalog.json documents, verifies them against the ARD spec and indexes the agents, MCP servers and skills they declare. Covers: [Agentic Resource Discovery](/protocols/ard/) · Last checked 2026-09-06 - [Discover](/category/discover/) [Neuronto ARD publisher list](https://neuronto.com/ard-publishers) Lists the domains it has found publishing an ARD manifest, each checked by fetching the manifest and testing endpoint reachability. Covers: [Agentic Resource Discovery](/protocols/ard/) · Last checked 2026-09-06 - [Pay](/category/pay/) [The Agent Almanac](https://agentalmanac.org/) Aggregates public numbers on agent services, payments and infrastructure across several commerce and agent protocols. Covers: [x402](/protocols/x402/), [Machine Payments Protocol](/protocols/mpp/), [Agentic Commerce Protocol](/protocols/agentic-commerce-protocol/), [Universal Commerce Protocol](/protocols/ucp/), [Agent Payments Protocol](/protocols/ap2/), [Agent2Agent Protocol](/protocols/a2a/), [Model Context Protocol](/protocols/mcp/) · Last checked 2026-09-06 - [Pay](/category/pay/) [x402 Foundation members](https://x402.org/members/) Lists the member organisations of the x402 Foundation. Covers: [x402](/protocols/x402/) · Last checked 2026-09-06 ## Registries - [Discover](/category/discover/) [AGNTCY Directory specification](https://spec.dir.agntcy.org/) Specification and API reference for the AGNTCY Agent Directory, a distributed registry of agent records. Covers: [Agent Directory Service](/protocols/ads/), [Open Agentic Schema Framework](/protocols/oasf/) · Last checked 2026-09-06 - [Act](/category/act/) [MCP Registry](https://github.com/modelcontextprotocol/registry) The Model Context Protocol project's own registry of published MCP servers, with its API and server list in the open. Covers: [Model Context Protocol](/protocols/mcp/) · Last checked 2026-09-06 ## Scanners - [Read](/category/read/) [Agent Ready](https://agent-ready.dev/) Scores a given site against the Vercel Agent Readability Spec and llmstxt.org and lists the fixes it found missing. Covers: [Agent Readability Spec](/protocols/vercel-agent-readability/), [llms.txt](/protocols/llms-txt/) · Last checked 2026-09-06 - [Read](/category/read/) [Lumar Agentic Readiness Scanner](https://agentic-readiness.lumar.io/) Checks which agent-facing technologies a site publishes — robots.txt, sitemaps, MCP, schema and Markdown twins — and whether each is implemented correctly. Covers: [llms.txt](/protocols/llms-txt/), [Model Context Protocol](/protocols/mcp/), [AI Crawler User-Agent Tokens in robots.txt](/protocols/ai-crawler-tokens-robots/) · Last checked 2026-09-06 ## Standards bodies - [Permit](/category/permit/) [IETF aipref working group](https://datatracker.ietf.org/wg/aipref/) The IETF working group page for AI preferences: charter, drafts and meeting materials. Covers: [AIPREF Vocabulary](/protocols/aipref-vocabulary/), [AIPREF Content-Usage Attachment](/protocols/aipref-content-usage-attachment/) · Last checked 2026-09-06 - [Identify](/category/identify/) [IETF webbotauth working group](https://datatracker.ietf.org/wg/webbotauth/) The IETF working group page for Web Bot Auth: charter, adopted drafts and meeting materials. Covers: [Web Bot Auth HTTP Message Signatures](/protocols/webbotauth/) · Last checked 2026-09-06 - [Identify](/category/identify/) [IETF wimse working group](https://datatracker.ietf.org/wg/wimse/) The IETF working group page for workload identity in multi-service environments: charter, drafts and meeting materials. Covers: [Workload Identity in Multi System Environments (WIMSE) Architecture](/protocols/wimse/) · Last checked 2026-09-06 ## Tools - [Permit](/category/permit/) [CrawlerCheck](https://crawlercheck.com/) Tests whether a site's robots.txt allows named search and AI crawlers to fetch a given URL. Covers: [AI Crawler User-Agent Tokens in robots.txt](/protocols/ai-crawler-tokens-robots/) · Last checked 2026-09-06 ## Trackers - [Act](/category/act/) [Agents Welcome — Agent Protocol Atlas](https://agentswelcome.dev/protocols) Groups agent-facing protocols by layer with links to each specification and to sample code. Covers: [Model Context Protocol](/protocols/mcp/), [WebMCP](/protocols/webmcp/), [Agent2Agent Protocol](/protocols/a2a/), [x402](/protocols/x402/), [Agent Payments Protocol](/protocols/ap2/), [Web Bot Auth HTTP Message Signatures](/protocols/webbotauth/), [llms.txt](/protocols/llms-txt/), [Really Simple Licensing](/protocols/rsl/), [agents.json](/protocols/agents-json/) · Last checked 2026-09-06 - [Act](/category/act/) [Chrome Platform Status: WebMCP](https://chromestatus.com/feature/5117755740913664) Tracks the WebMCP feature's implementation status in Chrome, including its origin trial and standards positions. Covers: [WebMCP](/protocols/webmcp/) · Last checked 2026-09-06 [JSON](/resources/index.json)