semgrep.dev agentic web readiness

A Manifest validity: 100% (1/1 valid, 1 security.txt error, 5 warnings) · Surface score 46 (coverage 30 + breadth 6 + conformance 10). 4 standards observed across 3 of 9 steps; 1 manifest present (+ robots.txt, security.txt), 1 security.txt error, 5 warnings. Checked 2026-10-25 20:26:48 UTC. This grades the agent-facing files published at semgrep.dev, not the organisation's products.

Badge for a README: ![agentic web grade](https://agenticweb.wiki/badge/semgrep.dev.svg) · JSON · Markdown · Re-check now

Coverage by step

Manifests

Contact: mailto:security@semgrep.dev
Expires: 2027-09-20T20:26:48Z
Canonical: https://semgrep.dev/.well-known/security.txt
Preferred-Languages: en
Policy: https://semgrep.dev/security-policy

What an agent can call

• MCP endpoint https://semgrep.dev/api/mcp (streamable-http) — from /.well-known/oauth-protected-resource
• llms.txt: Semgrep, 33 links
• robots.txt AI policy: no AI crawler tokens named; unlisted crawlers allowed
• OAuth protected resource https://semgrep.dev/api/mcp, authorization servers: https://login.semgrep.dev

Standards observed

• llms.txt https://semgrep.dev/llms.txt
• OAuth Protected Resource and Authorization Server Metadata (as used for agent authorization) https://semgrep.dev/.well-known/oauth-protected-resource
• OAuth Client ID Metadata Document https://semgrep.dev/.well-known/oauth-authorization-server
• Model Context Protocol

Cheapest next steps

1. Agentic Resource Discovery: a catalog at /.well-known/ard.json lets agents discover your APIs, MCP servers and data in one fetch (emerging spec — publish only if you run the underlying endpoints; generate via MCP template({"id": "ard", "domain": "semgrep.dev"}))
2. AGENTS.md: instructions for coding agents working on or with the site
   Generate starter /AGENTS.md for semgrep.dev

   # Semgrep for agents
   
   Semgrep and what it offers to agents
   
   ## Read the data, not the HTML
   - Index: https://semgrep.dev/llms.txt
   - API description: https://semgrep.dev/openapi.json
   - Discovery catalog: https://semgrep.dev/.well-known/ard.json
   
   ## Calling
   - MCP server (Streamable HTTP): https://semgrep.dev/mcp
   - Rate limits and authentication: describe them here
   
   ## Conventions
   - How to cite or attribute content
   - What not to do (no scraping of /account/, no automated sign-ups, etc.)
   
   Contact: security@semgrep.dev
   

3. api-catalog: A Well-Known URI and Link Relation to Help Discovery of APIs: an RFC 9727 linkset pointing at your OpenAPI description (conditional spec — publish only if you run the underlying endpoints; generate via MCP template({"id": "api-catalog-rfc9727", "domain": "semgrep.dev"}))

Agent pre-flight & practical readiness

Preferred mode: mcp (fallback llms_txt). Call the MCP endpoint at https://semgrep.dev/api/mcp first. Do not scrape the HTML when a tool can answer. Split-brain llms.txt: semgrep.dev/llms.txt has 33 links while docs.semgrep.dev/llms.txt has 125 technical links. Probe docs.semgrep.dev for full developer documentation.

Fetch: POST https://semgrep.dev/api/mcp POST a JSON-RPC tools/list, or server/discover on the 2026-07-28 revision, before reading HTML.

Subdomain discovery hint (split_brain_llms_txt): Split-brain llms.txt: semgrep.dev/llms.txt has 33 links while docs.semgrep.dev/llms.txt has 125 technical links. Probe docs.semgrep.dev for full developer documentation. (check docs.semgrep.dev)

Practical readability: exa_parsable yes, markdown negotiation no, render ssr_rich, HTML quality high, first content at byte 64702, HTML-to-llms.txt token ratio 16.08×.

AI crawler posture: open_to_all · Assistant access: OpenAI allowed, Anthropic allowed, Google allowed, Perplexity allowed, Exa allowed.

Readiness subscores: reach 20/20, readability 20/30, capabilities 18/35, conformance 10/15, total 68/100, grade A.

How the grade is made

+10 per lifecycle step with a standard observed, counting manifests that validate or that the probe rule for the standard recognises (a valid MCP server card counts for MCP; max 90); +3 per observed standard beyond the first two (max 15); conformance +15 (+10 with one manifest) minus 5 per agent-facing manifest with errors, never below 0 (security.txt is reported, not scored). Lifecycle grade: A+ >= 60, A >= 45, B >= 30, C >= 20, D >= 10. Strict 100-point composite (subscores.total_100 / subscores.grade_100): reach_and_policy (20) + readability (30) + capabilities (35) + conformance (15), where A+ >= 80, A >= 65, B >= 45, C >= 28, D >= 15. Errors are specification violations; warnings are best-practice gaps. Only public paths are fetched, without credentials, so standards that need a login (skipped in the probe) are not counted either way. The same report is the validate_manifest tool on the MCP server.

JSON · Markdown · probe rules