sitemap.xml vs llms.txt & ARD
sitemaps.org XML Sitemap (/sitemap.xml) is the classic web primitive built for human browsers and pre-integrated services (Enumerate every canonical HTML URL and lastmod timestamp so search-engine crawlers can index a site completely), whereas llms.txt and its step 1 (Discover) peers provide the machine-native agentic surface (/llms.txt) so autonomous agents can answer: "Where can an agent find a concise, curated map of a site's content, written for language models rather than browsers?" Across the agenticweb.wiki conformance panel, llms.txt is multi-implementation (307 of 574 probed origins (last checked 2026-10-23)).
How sitemap.xml compares to llms.txt & ARD at step 1 (Discover): why the classic primitive breaks for software agents, and how sites deploy both together.
| Dimension | sitemaps.org XML Sitemap (/sitemap.xml) Classic web | llms.txt Agentic web |
|---|---|---|
| Primary consumer | Human driving a web browser or pre-integrated developer code | Autonomous software agent, coding harness, or AI search/RAG crawler |
| Lifecycle step | Classic web baseline for step 1 (Discover) | 1 Discover: What is here, and where? |
| Core question answered | Enumerate every canonical HTML URL and lastmod timestamp so search-engine crawlers can index a site completely. | Where can an agent find a concise, curated map of a site's content, written for language models rather than browsers? |
| Wire format & entrypoint | /sitemap.xml (listed in /robots.txt) | well-known file at /llms.txt |
| Failure mode for agents | A sitemap.xml often lists tens of thousands of unranked URLs (marketing pages, pagination, tag archives) with zero descriptions, capability metadata, or links to Markdown twins—forcing an agent to guess which URLs matter. | Solved natively: llms.txt is a proposed convention for publishing a small, curated, markdown-formatted index of a site's content at /llms.txt (or a subpath such as /docs/llms.txt), intended to be read by language models and AI agents rather than crawled and scraped like a full site. |
| Governance & status | Established IETF / W3C / industry baseline | Jeremy Howard / Answer.AI; maintained as an open proposal in the AnswerDotAI/llms-txt GitHub repository with a public Discord for implementation discussion — "This is v2 of the proposal" — explicitly a community proposal open for input, not a standards-track document |
| Live adoption (/check/ census) | Universal across the classic web | multi-implementation — 307 of 574 probed origins (last checked 2026-10-23) |
| Primary specification | https://www.sitemaps.org/protocol.html | https://llmstxt.org |
Why sitemap.xml breaks when the caller is an agent
A sitemap.xml often lists tens of thousands of unranked URLs (marketing pages, pagination, tag archives) with zero descriptions, capability metadata, or links to Markdown twins—forcing an agent to guess which URLs matter.
When to use which (and how they work together)
- Keep sitemaps.org XML Sitemap (/sitemap.xml): You need exhaustive URL discovery for search indexers (Googlebot, Bingbot, OAI-SearchBot) across every public page on your origin.
- Add llms.txt: You want language models and agent harnesses to read a concise, curated Markdown index (/llms.txt) of your highest-signal documentation and a typed JSON-LD manifest (/.well-known/ard.json) of your callable MCP servers, A2A agents, and skills.
All agentic web standards replacing or extending sitemap.xml
- llms.txt — multi-implementation · 307/574 probed origins: Where can an agent find a concise, curated map of a site's content, written for language models rather than browsers?
- Agentic Resource Discovery (ARD) — multi-implementation · 19/576 probed origins: Where are an organization's discoverable agentic resources (MCP servers, A2A agents, skills, registries), and how can they be found by search rather than pre-installed one at a time?
- MCP Server Cards (SEP-2127) — multi-implementation · 19/19 probed origins: How can a client learn what an MCP server offers before connecting to it?
- api-catalog: A Well-Known URI and Link Relation to Help Discovery of APIs (RFC 9727) — multi-implementation · 93/574 probed origins: Given a publisher's domain, where is the machine-readable list of the APIs it exposes?
- agents.txt — prototype · 9/574 probed origins: Which agent-facing protocols (payments, auth, MCP, skills, A2A) does this site speak, and where?
- agents.json — prototype · 2/574 probed origins: How does an agent learn which multi-step API call sequences ("flows") a service supports, on top of its raw OpenAPI surface?