# llms.txt

[4 · Read](/category/read/)

| llms.txt |
| --- |
| Category | [Read](/category/read/) |
| Governed by | Jeremy Howard / Answer.AI; maintained as an open proposal in the AnswerDotAI/llms-txt GitHub repository with a public Discord for implementation discussion |
| Status | "This is v2 of the proposal" — explicitly a community proposal open for input, not a standards-track document |
| Phase | draft |
| Implementation | multi-implementation |
| Adoption | claimed — Rankability (Tranco top-1000, June 2026: 8.7%) and SE Ranking (~300,000 domains analyzed: 10.13%) — independent third-party publisher-side counts, not our own probe |
| Version | v2 (2026-08) |
| Specification | [https://llmstxt.org](https://llmstxt.org) |
| Found at | /llms.txt (or /<path>/llms.txt, e.g. /docs/llms.txt) |
| Last checked | 2026-09-06 |

[Read](/category/read/) llms.txt is a proposed convention for publishing a small, curated, markdown-formatted index of a site's content at /llms.txt (or a subpath such as /docs/llms.txt), intended to be read by language models and AI agents rather than crawled and scraped like a full site [[1]](#ref-1). The file is one required H1 heading naming the project, an optional blockquote summary, optional free-form context, and then H2-delimited "file lists" of markdown links to more detailed content — typically clean .md mirrors of a site's HTML pages [[1]](#ref-1).

## Overview
It solves a narrow problem: HTML pages are built for browsers, wrap information in navigation and scripts that are wasteful to parse into an LLM's limited context window, and give an agent no small, authoritative starting point for a large site. llms.txt gives a site a place to say "start here" and to link to markdown-native content instead of full HTML [[1]](#ref-1).

It does not solve discovery for sites that don't publish one, does not replace robots.txt (a permission signal, not a content index) or sitemap.xml (exhaustive, for search-engine crawlers, and usually not markdown-native) [[1]](#ref-1), and — critically — it does not guarantee that anything reads it. Google's own Search Central documentation, last updated 2026-07-10, explicitly tells site owners they "don't need to create ... llms.txt" because "Google Search itself doesn't use them," filing it alongside other AEO/GEO "hacks" to ignore [[3]](#ref-3). No answer engine has published documentation describing llms.txt consumption in its retrieval pipeline.

The current text is v2, published August 2026, revising the original September 2024 proposal by Jeremy Howard (Answer.AI). v2's main addition is standard discovery link relations (rel="alternate" type="text/markdown" for a page's markdown twin, rel="describedby" for the covering llms.txt file), expressible as HTML <link> tags or an HTTP Link: header; it also loosened the markdown-mirror URL convention and dropped the v1 context-expansion tooling and the special meaning of the "Optional" section [[1]](#ref-1)[[2]](#ref-2). The specification remains an informal, single-maintainer GitHub proposal, not a standards-track document of any kind [[1]](#ref-1).

On the producer side, several documentation platforms and CMSs generate llms.txt automatically — Mintlify, GitBook, Yoast SEO, AIOSEO, and Wix — and OpenAI, Anthropic, and Google publish llms.txt for their own developer docs [[1]](#ref-1). This is genuine, if modest, publisher-side adoption: independent Tranco-based studies put it at roughly 8.7% of the top 1,000 domains (June 2026) and 5.6% of the top 10,000 [[4]](#ref-4), while a broader ~300,000-domain analysis found 10.13% overall, spread fairly evenly across traffic tiers rather than concentrated among high-traffic sites [[5]](#ref-5).

What is disputed or unknown is consumption, not existence. The gap between "a file exists" and "a model reads it" is large and, per the sources available, mostly unmeasured by anyone with visibility into model retrieval internals. One large-scale server-log study of 137,000 llms.txt-publishing sites found 97% of the files were never fetched by any client during the observation window, and of the requests that did arrive, 96% came from generic bots rather than identifiable AI assistants [[6]](#ref-6). Combined with Google's explicit statement that its own systems ignore the file, the safest characterization is: llms.txt is a real, growing publisher-side convention with no documented reader among major answer engines as of mid-2026.

## Discovery and probe

Fixed filename at the site root or at any subpath, covering the pages under that path · /llms.txt (or /<path>/llms.txt, e.g. /docs/llms.txt)

```
GET {origin}/llms.txt
parse:     200 response beginning with an H1 heading (`# `) followed by a blockquote summary, per the llms.txt v2 grammar
on absent: No llms.txt at this path; a more specific subpath file (e.g. /docs/llms.txt) may still exist and take precedence for that subpath
```

Live check run by [the MCP probe](/mcp/server-card): GET /llms.txt. llms.txt must begin with an H1 line.

```
{
 "path": "/llms.txt",
 "expect": {
  "text_match": "(?m)^# \\S"
 },
 "note": "llms.txt must begin with an H1 line."
}
```

## History
- 2026-09-05 — seeded from Exa Agent research run
- 2026-09-06 — verified against llmstxt.org (v2, Aug 2026), llmstxt.org/changes.html, Google's AI optimization guide, and three independent third-party adoption studies (Rankability, SE Ranking, Ahrefs)

## Instances

- [Yoast SEO (WordPress plugin, generates and maintains llms.txt)](https://yoast.com/features/llms-txt/) · Recorded 2026-09-06 · site
- [Wix (generates an llms.txt for every Wix site)](https://support.wix.com/en/article/understanding-your-sites-llmstxt-file) · Recorded 2026-09-06 · site
- [OpenAI developer docs llms.txt](https://developers.openai.com/llms.txt) · Recorded 2026-09-06 · site
- [Mintlify (auto-generates llms.txt and markdown page twins for hosted docs sites)](https://www.mintlify.com/docs/ai/llmstxt) · Recorded 2026-09-06 · site
- [Google Gemini API docs llms.txt](https://ai.google.dev/gemini-api/docs/llms.txt) · Recorded 2026-09-06 · site
- [GitBook (serves an llms.txt for published docs sites)](https://www.gitbook.com/blog/what-is-llms-txt) · Recorded 2026-09-06 · site
- [FastHTML / Answer.AI docs (reference implementation cited by the spec itself)](https://www.fastht.ml/docs/llms.txt) · Recorded 2026-09-06 · site
- [Anthropic docs llms.txt](https://docs.anthropic.com/llms.txt) · Recorded 2026-09-06 · site
- [AIOSEO (WordPress plugin with an llms.txt generator)](https://aioseo.com/features/llms-txt/) · Recorded 2026-09-06 · site
- [agenticweb.wiki](https://agenticweb.wiki/llms.txt) · Observed 2026-09-06 · site
- [Alibaba Cloud](https://www.alibabacloud.com) · Observed 2026-07-16 · business
- [n8n](https://docs.n8n.io) · Observed 2026-06-05 · business
- [agent-ready-mcp](https://github.com/mlava/agent-ready-mcp) · Observed 2026-05-11 · service

Every instance with its source is listed on the [instances page](/instances/).

## See also

## References

- Full v2 proposal text: file grammar, discovery link relations (rel="alternate" type="text/markdown", rel="describedby"), relationship to robots.txt/sitemap.xml, and that AI labs (OpenAI, Anthropic, Gemini) publish llms.txt for their own docs — [https://llmstxt.org](https://llmstxt.org) (2026-08) VERIFIED
- v2 (August 2026) changelog: adds standard link relations for discoverability, allows both `page.html.md` and `page.md` forms, defines what a subpath file covers, and drops the v1 context-expansion tooling and the mechanical meaning of the "Optional" section — [https://llmstxt.org/changes.html](https://llmstxt.org/changes.html) (2026-08) VERIFIED
- Google's official Search Central guidance (last updated 2026-07-10) lists "LLMS.txt files and other 'special' markup" under things site owners can ignore, stating Google Search's generative AI features do not use them — [https://developers.google.com/search/docs/fundamentals/ai-optimization-guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) (2026-07-10) VERIFIED
- 8.7% of the Tranco top-1,000 domains and 5.6% of the top-10,000 published a valid llms.txt file as of June 2026 — [https://www.rankability.com/data/llms-txt-adoption/](https://www.rankability.com/data/llms-txt-adoption/) (2026-06) REPORTED
- Across a dataset of nearly 300,000 domains, 10.13% carried an llms.txt file, with adoption not concentrated among high-traffic sites (8.27% for 100,001+ visit sites vs 10.54% for mid-traffic sites) — [https://seranking.com/blog/llms-txt/](https://seranking.com/blog/llms-txt/) (2026) REPORTED
- Analysis of 137,000 sites publishing llms.txt found 97% of the files were never fetched by any client in the observation window; of requests that did reach a file, 96% came from bots rather than identifiable AI assistants — [https://ahrefs.com/blog/llmstxt-study/](https://ahrefs.com/blog/llmstxt-study/) (2026) REPORTED

Disputed: Publisher-side adoption (thousands of sites, roughly 8-10% of measured domain samples) is separate from and should not be conflated with consumption: no answer engine has documented reading llms.txt, Google explicitly says its generative AI features don't use it, and a large-scale log analysis found the great majority of published files are never fetched.

[JSON](/data/protocols/llms-txt.json) · [Markdown](/protocols/llms-txt/index.md)