mcp.apify.com agentic web readiness

B Manifest validity: 100% (1/1 valid, 0 errors, 6 warnings) · Surface score 33 (coverage 20 + breadth 3 + conformance 10). 3 standards observed across 2 of 9 steps; 1 manifest present, 0 errors, 6 warnings. Checked 2026-10-25 22:05:17 UTC. This grades the agent-facing files published at mcp.apify.com, not the organisation's products.

Badge for a README: ![agentic web grade](https://agenticweb.wiki/badge/mcp.apify.com.svg) · JSON · Markdown · Re-check now

Coverage by step

Manifests

What an agent can call

• MCP endpoint https://mcp.apify.com/ (streamable-http), tools: search-actors, fetch-actor-details, call-actor, get-actor-run, get-dataset-items, get-key-value-store-record, abort-actor-run, search-apify-docs, fetch-apify-docs — from /.well-known/mcp/server-card.json
• MCP endpoint https://mcp.apify.com (streamable-http) — from /.well-known/oauth-protected-resource
• OAuth protected resource https://mcp.apify.com, authorization servers: https://console-backend.apify.com
• Endpoint check (mcp https://mcp.apify.com/): pseudo_endpoint_html (MCP endpoint URL points to the HTML homepage rather than a JSON-RPC endpoint)
• Endpoint check (mcp https://mcp.apify.com): pseudo_endpoint_html (MCP endpoint URL points to the HTML homepage rather than a JSON-RPC endpoint)

Standards observed

• OAuth Protected Resource and Authorization Server Metadata (as used for agent authorization) https://mcp.apify.com/.well-known/oauth-protected-resource
• OAuth Client ID Metadata Document https://mcp.apify.com/.well-known/oauth-authorization-server
• MCP Server Cards (SEP-2127)

Cheapest next steps

1. llms.txt: one Markdown index file; the most widely read agent-facing file
   Generate starter /llms.txt for mcp.apify.com

   # Mcp
   
   > Mcp and what it offers to agents
   
   Short notes an agent should know before using the site: what it is for, who runs it, how to cite it.
   
   ## Docs
   - [Getting started](https://mcp.apify.com/docs/getting-started): the first page to read
   - [API reference](https://mcp.apify.com/openapi.json): OpenAPI description
   
   ## Optional
   - [Changelog](https://mcp.apify.com/changelog): what changed, newest first
   

2. AI Crawler User-Agent Tokens in robots.txt: state what AI crawlers may do in robots.txt you already serve
   Generate starter /robots.txt for mcp.apify.com

   # search and answer-engine crawlers
   User-agent: Googlebot
   User-agent: bingbot
   User-agent: OAI-SearchBot
   User-agent: Claude-SearchBot
   User-agent: PerplexityBot
   User-agent: ExaSearchBot
   User-agent: TavilyBot
   Allow: /
   
   # model-training crawlers: choose Allow or Disallow and keep it explicit
   User-agent: GPTBot
   User-agent: ClaudeBot
   User-agent: Google-Extended
   User-agent: Applebot-Extended
   User-agent: CCBot
   User-agent: meta-externalagent
   User-agent: Bytespider
   Disallow: /
   
   # user-triggered fetchers (a person asked for this page)
   User-agent: ChatGPT-User
   User-agent: Claude-User
   User-agent: Perplexity-User
   Allow: /
   
   # default fallback for unlisted crawlers (RFC 9309 §2.2.1: specific groups above take precedence over *)
   User-agent: *
   Allow: /
   Disallow: /account/
   Content-Signal: ai-train=no, search=yes, ai-input=yes
   
   Sitemap: https://mcp.apify.com/sitemap.xml
   Agentmap: https://mcp.apify.com/.well-known/ard.json
   

3. Agentic Resource Discovery: a catalog at /.well-known/ard.json lets agents discover your APIs, MCP servers and data in one fetch (emerging spec — publish only if you run the underlying endpoints; generate via MCP template({"id": "ard", "domain": "mcp.apify.com"}))

Agent pre-flight & practical readiness

Preferred mode: exa_or_http_fetch. No structured agent endpoint was found, and the page is not a readable document. Prefer a search index over raw HTML.

Fetch: GET https://mcp.apify.com/ No structured agent endpoint was found. Fetch the public page or use a search index.

Practical readability: exa_parsable no, markdown negotiation no, render csr_empty_shell, HTML quality js_shell.

AI crawler posture: unspecified · Assistant access: OpenAI unspecified, Anthropic unspecified, Google unspecified, Perplexity unspecified, Exa unspecified.

Readiness subscores: reach 0/20, readability 0/30, capabilities 2/35, conformance 10/15, total 12/100, grade F.

How the grade is made

+10 per lifecycle step with a standard observed, counting manifests that validate or that the probe rule for the standard recognises (a valid MCP server card counts for MCP; max 90); +3 per observed standard beyond the first two (max 15); conformance +15 (+10 with one manifest) minus 5 per agent-facing manifest with errors, never below 0 (security.txt is reported, not scored). Lifecycle grade: A+ >= 60, A >= 45, B >= 30, C >= 20, D >= 10. Strict 100-point composite (subscores.total_100 / subscores.grade_100): reach_and_policy (20) + readability (30) + capabilities (35) + conformance (15), where A+ >= 80, A >= 65, B >= 45, C >= 28, D >= 15. Errors are specification violations; warnings are best-practice gaps. Only public paths are fetched, without credentials, so standards that need a login (skipped in the probe) are not counted either way. The same report is the validate_manifest tool on the MCP server.

JSON · Markdown · probe rules