scrapfly.io agentic web readiness

A+ Manifest validity: 83% (5/6 valid, 4 errors, 12 warnings) · Surface score 75 (coverage 50 + breadth 15 + conformance 10). 9 standards observed across 5 of 9 steps; 6 manifests present (+ security.txt), 4 errors, 12 warnings. Checked 2026-10-25 22:05:23 UTC. This grades the agent-facing files published at scrapfly.io, not the organisation's products.

Badge for a README: ![agentic web grade](https://agenticweb.wiki/badge/scrapfly.io.svg) · JSON · Markdown · Re-check now

Coverage by step

Manifests

{
  "$schema": "https://static.modelcontextprotocol.io/schemas/v1/server-card.schema.json",
  "name": "io.scrapfly/scrapfly",
  "title": "Scrapfly",
  "version": "1.0.0",
  "description": "Scrapfly and what it offers to agents",
  "websiteUrl": "https://scrapfly.io/",
  "icons": [
    {
      "src": "https://scrapfly.io/icon.png",
      "mimeType": "image/png",
      "sizes": [
        "180x180"
      ]
    }
  ],
  "remotes": [
    {
      "type": "streamable-http",
      "url": "https://scrapfly.io/mcp",
      "supportedProtocolVersions": [
        "2025-11-25",
        "2025-06-18",
        "2025-03-26"
      ]
    }
  ]
}

What an agent can call

• MCP endpoint https://mcp.scrapfly.io (streamable-http), tools: scrape_url, extract_data, take_screenshot, crawl_site, browser_action — from /.well-known/mcp.json
• MCP endpoint https://mcp.scrapfly.io/mcp (streamable-http) — from /.well-known/mcp/server-card.json
• A2A agent scrapfly at https://mcp.scrapfly.io, skills: Scrape a URL, Extract structured data, Take a screenshot, Crawl a site, Cloud Browser action
• API description https://scrapfly.io/docs/scrape-api/getting-started (text/html)
• API description https://scrapfly.io/docs/scrape-api/getting-started?view=markdown (text/markdown)
• API description https://scrapfly.io/api_errors.json (application/json)
• API description https://scrapfly.io/docs/scrape-api/errors (text/html)
• API description https://mcp.scrapfly.io/.well-known/mcp/server-card.json (application/json)
• API description https://scrapfly.io/docs/mcp (text/html)
• llms.txt: Scrapfly, 0 links
• robots.txt AI policy: blocks CCBot, meta-externalagent, Bytespider, Amazonbot, cohere-ai, Diffbot; unlisted crawlers partial
• OAuth protected resource https://mcp.scrapfly.io, authorization servers: https://mcp.scrapfly.io

Standards observed

• Agent2Agent Protocol https://scrapfly.io/.well-known/agent-card.json
• llms.txt https://scrapfly.io/llms.txt
• api-catalog: A Well-Known URI and Link Relation to Help Discovery of APIs https://scrapfly.io/.well-known/api-catalog
• AI Crawler User-Agent Tokens in robots.txt https://scrapfly.io/robots.txt
• OAuth Protected Resource and Authorization Server Metadata (as used for agent authorization) https://scrapfly.io/.well-known/oauth-protected-resource
• Agent Skills https://scrapfly.io/.well-known/agent-skills/index.json
• Markdown for Agents https://scrapfly.io/
• MCP Server Cards (SEP-2127)
• Model Context Protocol

Cheapest next steps

1. Agentic Resource Discovery: a catalog at /.well-known/ard.json lets agents discover your APIs, MCP servers and data in one fetch (emerging spec — publish only if you run the underlying endpoints; generate via MCP template({"id": "ard", "domain": "scrapfly.io"}))
2. AGENTS.md: instructions for coding agents working on or with the site
   Generate starter /AGENTS.md for scrapfly.io

   # Scrapfly for agents
   
   Scrapfly and what it offers to agents
   
   ## Read the data, not the HTML
   - Index: https://scrapfly.io/llms.txt
   - API description: https://scrapfly.io/openapi.json
   - Discovery catalog: https://scrapfly.io/.well-known/ard.json
   
   ## Calling
   - MCP server (Streamable HTTP): https://scrapfly.io/mcp
   - Rate limits and authentication: describe them here
   
   ## Conventions
   - How to cite or attribute content
   - What not to do (no scraping of /account/, no automated sign-ups, etc.)
   
   Contact: security@scrapfly.io
   

Agent pre-flight & practical readiness

Preferred mode: mcp (fallback openapi). Call the MCP endpoint at https://mcp.scrapfly.io/mcp first. Do not scrape the HTML when a tool can answer.

Fetch: POST https://mcp.scrapfly.io/mcp POST a JSON-RPC tools/list, or server/discover on the 2026-07-28 revision, before reading HTML.

Practical readability: exa_parsable yes, markdown negotiation yes, render markdown_native, HTML quality high, first content at byte 0, HTML-to-llms.txt token ratio 3.202×.

AI crawler posture: search_allowed_training_blocked · Assistant access: OpenAI partial, Anthropic partial, Google partial, Perplexity partial, Exa partial.

Readiness subscores: reach 18/20, readability 30/30, capabilities 35/35, conformance 10/15, total 93/100, grade A+.

How the grade is made

+10 per lifecycle step with a standard observed, counting manifests that validate or that the probe rule for the standard recognises (a valid MCP server card counts for MCP; max 90); +3 per observed standard beyond the first two (max 15); conformance +15 (+10 with one manifest) minus 5 per agent-facing manifest with errors, never below 0 (security.txt is reported, not scored). Lifecycle grade: A+ >= 60, A >= 45, B >= 30, C >= 20, D >= 10. Strict 100-point composite (subscores.total_100 / subscores.grade_100): reach_and_policy (20) + readability (30) + capabilities (35) + conformance (15), where A+ >= 80, A >= 65, B >= 45, C >= 28, D >= 15. Errors are specification violations; warnings are best-practice gaps. Only public paths are fetched, without credentials, so standards that need a login (skipped in the probe) are not counted either way. The same report is the validate_manifest tool on the MCP server.

JSON · Markdown · probe rules