Useful resources

Scanners, registries, indexes and standards-body pages that tell you something checkable about the agentic web. Each entry says what it does and which articles it covers.

Documentation

• Permit Cloudflare AI Crawl Control bot reference

  Names the AI crawlers Cloudflare recognises, with the operator, user-agent token and stated purpose of each.

  Covers: AI Crawler User-Agent Tokens in robots.txt · Last checked 2026-09-06

• Identify Cloudflare Web Bot Auth documentation

  Documents how Cloudflare verifies HTTP message signatures from bots at the edge and how a bot operator registers keys.

  Covers: Web Bot Auth HTTP Message Signatures, HTTP Message Signatures · Last checked 2026-09-06

• Read llmstxt.org

  The llms.txt proposal's own site: the file format, its rationale and a directory of sites publishing one.

  Covers: llms.txt · Last checked 2026-09-06

Indexes

• Discover ardvark

  Crawls the web for ai-catalog.json documents, verifies them against the ARD spec and indexes the agents, MCP servers and skills they declare.

  Covers: Agentic Resource Discovery · Last checked 2026-09-06

• Discover Neuronto ARD publisher list

  Lists the domains it has found publishing an ARD manifest, each checked by fetching the manifest and testing endpoint reachability.

  Covers: Agentic Resource Discovery · Last checked 2026-09-06

• Pay The Agent Almanac

  Aggregates public numbers on agent services, payments and infrastructure across several commerce and agent protocols.

  Covers: x402, Machine Payments Protocol, Agentic Commerce Protocol, Universal Commerce Protocol, Agent Payments Protocol, Agent2Agent Protocol, Model Context Protocol · Last checked 2026-09-06

• Pay x402 Foundation members

  Lists the member organisations of the x402 Foundation.

  Covers: x402 · Last checked 2026-09-06

Registries

• Discover AGNTCY Directory specification

  Specification and API reference for the AGNTCY Agent Directory, a distributed registry of agent records.

  Covers: Agent Directory Service, Open Agentic Schema Framework · Last checked 2026-09-06

• Act MCP Registry

  The Model Context Protocol project's own registry of published MCP servers, with its API and server list in the open.

  Covers: Model Context Protocol · Last checked 2026-09-06

Scanners

• Read Agent Ready

  Scores a given site against the Vercel Agent Readability Spec and llmstxt.org and lists the fixes it found missing.

  Covers: Agent Readability Spec, llms.txt · Last checked 2026-09-06

• Read Lumar Agentic Readiness Scanner

  Checks which agent-facing technologies a site publishes — robots.txt, sitemaps, MCP, schema and Markdown twins — and whether each is implemented correctly.

  Covers: llms.txt, Model Context Protocol, AI Crawler User-Agent Tokens in robots.txt · Last checked 2026-09-06

Standards bodies

• Permit IETF aipref working group

  The IETF working group page for AI preferences: charter, drafts and meeting materials.

  Covers: AIPREF Vocabulary, AIPREF Content-Usage Attachment · Last checked 2026-09-06

• Identify IETF webbotauth working group

  The IETF working group page for Web Bot Auth: charter, adopted drafts and meeting materials.

  Covers: Web Bot Auth HTTP Message Signatures · Last checked 2026-09-06

• Identify IETF wimse working group

  The IETF working group page for workload identity in multi-service environments: charter, drafts and meeting materials.

  Covers: Workload Identity in Multi System Environments (WIMSE) Architecture · Last checked 2026-09-06

Tools

• Permit CrawlerCheck

  Tests whether a site's robots.txt allows named search and AI crawlers to fetch a given URL.

  Covers: AI Crawler User-Agent Tokens in robots.txt · Last checked 2026-09-06

Trackers

• Act Agents Welcome — Agent Protocol Atlas

  Groups agent-facing protocols by layer with links to each specification and to sample code.

  Covers: Model Context Protocol, WebMCP, Agent2Agent Protocol, x402, Agent Payments Protocol, Web Bot Auth HTTP Message Signatures, llms.txt, Really Simple Licensing, agents.json · Last checked 2026-09-06

• Act Chrome Platform Status: WebMCP

  Tracks the WebMCP feature's implementation status in Chrome, including its origin trial and standards positions.

  Covers: WebMCP · Last checked 2026-09-06

JSON