---
type: Agentic Web Standard
title: "llms.txt"
description: "Where can an agent find a concise, curated map of a site's content, written for language models rather than browsers?"
resource: "https://llmstxt.org"
tags: ["read", "document-format", "community", "serve-content-to-agents", "discover-my-api"]
status: draft
generated: { by: agenticweb-builder/1.0, at: 2026-10-30T00:00:00Z }
verified: { by: process:agenticweb-engine, at: 2026-10-28T00:00:00Z }
stale_after: 2026-11-27T00:00:00Z
sources:
  - id: ref-1
    resource: "https://llmstxt.org"
    title: "Howard, J., The /llms.txt file, v2 (published 2024-09-03; modified 2026-08-10): \"We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content. The file can be placed at the site root, or at any path within it, covering the pages under that path. This file offers brief background information, guidance, and links to detailed markdown files. … We furthermore propose that pages with information that agents might need provide a clean markdown version of those pages at the same URL as the original page, either with .md appended (page.html.md) or with the extension replaced by .md (page.md). (URLs without file names should append index.html.md or index.md instead.) To help clients find these files, we recommend using standard link relations: rel=\"alternate\" type=\"text/markdown\" points to the markdown version of a page, and rel=\"describedby\" points to the llms.txt file that covers it. … Agents are expected to view or search llms.txt to find the information they need, then follow the relevant links. … An H1 with the name of the project or site. This is the only required section … While sitemaps list all pages for search engines, llms.txt offers a curated overview for LLMs. … robots.txt lets automated tools know what access to a site is considered acceptable, such as for search indexing bots. llms.txt information is instead used on demand, when an agent needs information about a topic while assisting a user.\""
    last_modified: 2026-08-10T00:00:00Z
  - id: ref-2
    resource: "https://llmstxt.org/changes.html"
    title: "Howard, J., Changes (llmstxt.org, v2, August 2026): \"v2 instead states the expectation directly: agents view or search the llms.txt to find what they need, then follow the relevant links, which should point to LLM-friendly content. The context-expansion tooling is no longer part of the proposal, and with it goes the special meaning of the Optional section, which told those tools what to omit. Optional sections are still allowed, and remain a useful convention for secondary links, but they no longer carry mechanical semantics.\""
    last_modified: 2026-08-10T00:00:00Z
  - id: ref-3
    resource: "https://developers.google.com/search/docs/fundamentals/ai-optimization-guide"
    title: "Google Search Central, Optimizing your website for generative AI features on Google Search (updated 2026-07-10): \"You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them.\""
    last_modified: 2026-07-10T00:00:00Z
  - id: ref-4
    resource: "https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt"
    title: "Chrome for Developers, llms.txt (Lighthouse, updated 2026-05-05): \"The llms.txt file is an emerging convention used to provide a machine-readable summary of a website's content, specifically designed for LLMs and AI agents. … Lighthouse flags the pages if a server error occurs when attempting to retrieve the llms.txt file. If the file is not provided by the server (resulting in a 404), the audit is marked as Not Applicable (N/A), as providing the file is optional at the moment.\""
    last_modified: 2026-05-05T00:00:00Z
  - id: ref-5
    resource: "https://www.rankability.com/blog/llms-txt-adoption/"
    title: "Rankability, llms.txt Adoption Research Report (2026-09-18): \"9.3% of the top 1,000 domains serve a confirmed llms.txt or llms-full.txt file.\""
    last_modified: 2026-09-18T00:00:00Z
  - id: ref-6
    resource: "https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents"
    title: "Vercel, Make your site readable by AI agents (Knowledge Base, with ora.ai, 2026-08-28): \"Of the 329 runs that reached it, 325 had a scorable steering result. Of those 325 runs, 117, or 36%, later fetched a page it listed. Across all 329 reaching runs, 120, or 36%, grounded the final answer in its content.\""
    last_modified: 2026-08-28T00:00:00Z
  - id: ref-7
    resource: "https://ahrefs.com/blog/llmstxt-study/"
    title: "Ahrefs, We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read (2026-06-15; data: May 2026): \"28% of the 137K domains using Ahrefs Web Analytics publish an llms.txt file. 97% of those files received zero traffic in May 2026.\""
    last_modified: 2026-06-15T00:00:00Z
---

# llms.txt

llms.txt is an agentic-web standard (document format — /llms.txt) at step 4, Read, governed by Jeremy Howard; its status is "Community proposal, not standards-track". It answers: "Where can an agent find a concise, curated map of a site's content, written for language models rather than browsers?" Replacing or extending sitemap.xml / a README for a site's documentation, implementation status is multi-implementation, last verified 2026-10-28.

# Overview

llms.txt is a proposal by Jeremy Howard (published 2024-09-03, revised to v2 on 2026-08-10) to add an `/llms.txt` Markdown file at the root of a website or any subpath (such as `/docs/llms.txt`) providing brief background information, guidance, and links to detailed Markdown files for language models [^ref-1]. Where `sitemap.xml` lists all indexable pages for search engines (and does not include URLs to external sites), `llms.txt` curates a concise overview for agents viewing or searching `llms.txt` and following links on demand [^ref-1][^ref-2]; and unlike `robots.txt`, which tells automated tools what access is acceptable, `llms.txt` is used on demand when an agent needs information [^ref-1].

A conforming `llms.txt` v2 file places its sections in a defined order: an optional byte-order mark (BOM), a required `H1` heading (`# Project or Site Name`, the only required section), an optional summary blockquote (`> ...`), zero or more body sections of any type except headings (such as paragraphs or lists), and zero or more `H2` sections (`## Section Name`) containing Markdown lists of `[name](url)` links with optional `: notes` [^ref-1] (`## Optional` remains a convention for secondary links but carries no mechanical semantics in v2 [^ref-2]). The spec also proposes that pages agents might need provide clean Markdown versions at the same URL with `.md` appended (`page.html.md`) or replacing the extension (`page.md`), advertised via HTML `<link>` elements or HTTP `Link:` headers (`rel="alternate" type="text/markdown"` for pages and `rel="describedby"` for the covering `llms.txt`) [^ref-1][^ref-2].

# Examples

```http
GET {origin}/llms.txt
# parse: 200 response beginning with an H1 heading (`# <project/site name>`) followed by a blockquote summary, per the llms.txt v2 grammar
# on absent: No llms.txt at this path; a more specific subpath file (e.g. /docs/llms.txt) may still exist and take precedence for that subpath
```

# Knowledge Graph

* **Lifecycle Step**: [4 Read](/category/read/index.md)
* **Pairs with**: [AGENTS.md](/okf/protocols/agents-md.md), [Agent Readability Spec](/okf/protocols/vercel-agent-readability.md), [AI Catalog well-known file](/okf/protocols/ai-catalog-well-known-file.md), [NLWeb](/okf/protocols/nlweb.md), [AI crawler tokens in robots.txt](/okf/protocols/ai-crawler-tokens-robots.md), [Agent Skills](/okf/protocols/anthropic-agent-skills-format.md)
* **Related terms**: [llms.txt file (term)](/okf/terms/llms-txt-file.md), [Markdown twin](/okf/terms/markdown-twin.md), [Context compaction](/okf/terms/context-compaction.md), [Coding agent instruction files](/okf/terms/coding-agent-instructions.md)
* **Compared in**: [sitemap.xml vs llms.txt & ARD (classic vs agentic)](/compare/sitemap-xml-vs-llms-txt/index.md), [llms.txt vs AGENTS.md](/compare/llms-txt-vs-agents-md/index.md), [Agentic Resource Discovery (ARD) vs llms.txt](/compare/ard-vs-llms-txt/index.md), [Agent Readability Spec vs llms.txt](/compare/vercel-agent-readability-vs-llms-txt/index.md), [llms.txt vs agents.txt](/compare/llms-txt-vs-agents-txt/index.md), [Open Knowledge Format vs llms.txt](/compare/okf-vs-llms-txt/index.md), [Markdown for Agents vs llms.txt](/compare/cloudflare-markdown-for-agents-vs-llms-txt/index.md), [Agent Skills vs llms.txt](/compare/anthropic-agent-skills-format-vs-llms-txt/index.md)

[^ref-1]: Howard, J., The /llms.txt file, v2 (published 2024-09-03; modified 2026-08-10): "We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content. The file can be placed at the site root, or at any path within it, covering the pages under that path. This file offers brief background information, guidance, and links to detailed markdown files. … We furthermore propose that pages with information that agents might need provide a clean markdown version of those pages at the same URL as the original page, either with .md appended (page.html.md) or with the extension replaced by .md (page.md). (URLs without file names should append index.html.md or index.md instead.) To help clients find these files, we recommend using standard link relations: rel="alternate" type="text/markdown" points to the markdown version of a page, and rel="describedby" points to the llms.txt file that covers it. … Agents are expected to view or search llms.txt to find the information they need, then follow the relevant links. … An H1 with the name of the project or site. This is the only required section … While sitemaps list all pages for search engines, llms.txt offers a curated overview for LLMs. … robots.txt lets automated tools know what access to a site is considered acceptable, such as for search indexing bots. llms.txt information is instead used on demand, when an agent needs information about a topic while assisting a user." (https://llmstxt.org)
[^ref-2]: Howard, J., Changes (llmstxt.org, v2, August 2026): "v2 instead states the expectation directly: agents view or search the llms.txt to find what they need, then follow the relevant links, which should point to LLM-friendly content. The context-expansion tooling is no longer part of the proposal, and with it goes the special meaning of the Optional section, which told those tools what to omit. Optional sections are still allowed, and remain a useful convention for secondary links, but they no longer carry mechanical semantics." (https://llmstxt.org/changes.html)
[^ref-3]: Google Search Central, Optimizing your website for generative AI features on Google Search (updated 2026-07-10): "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." (https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)
[^ref-4]: Chrome for Developers, llms.txt (Lighthouse, updated 2026-05-05): "The llms.txt file is an emerging convention used to provide a machine-readable summary of a website's content, specifically designed for LLMs and AI agents. … Lighthouse flags the pages if a server error occurs when attempting to retrieve the llms.txt file. If the file is not provided by the server (resulting in a 404), the audit is marked as Not Applicable (N/A), as providing the file is optional at the moment." (https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt)
[^ref-5]: Rankability, llms.txt Adoption Research Report (2026-09-18): "9.3% of the top 1,000 domains serve a confirmed llms.txt or llms-full.txt file." (https://www.rankability.com/blog/llms-txt-adoption/)
[^ref-6]: Vercel, Make your site readable by AI agents (Knowledge Base, with ora.ai, 2026-08-28): "Of the 329 runs that reached it, 325 had a scorable steering result. Of those 325 runs, 117, or 36%, later fetched a page it listed. Across all 329 reaching runs, 120, or 36%, grounded the final answer in its content." (https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents)
[^ref-7]: Ahrefs, We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read (2026-06-15; data: May 2026): "28% of the 137K domains using Ahrefs Web Analytics publish an llms.txt file. 97% of those files received zero traffic in May 2026." (https://ahrefs.com/blog/llmstxt-study/)