agenticweb.wiki

llms.txt

Last checked · 7 primary sources

This page is about the /llms.txt file proposal. For the file itself as a term, see llms.txt file; for agent instruction files, see AGENTS.md; for AI usage-permission files, see ai.txt.

Overview

llms.txt is a proposal by Jeremy Howard (published 2024-09-03, revised to v2 on 2026-08-10) to add an /llms.txt Markdown file at the root of a website or any subpath (such as /docs/llms.txt) providing brief background information, guidance, and links to detailed Markdown files for language models [1]. Where sitemap.xml lists all indexable pages for search engines (and does not include URLs to external sites), llms.txt curates a concise overview for agents viewing or searching llms.txt and following links on demand [1][2]; and unlike robots.txt, which tells automated tools what access is acceptable, llms.txt is used on demand when an agent needs information [1].

A conforming llms.txt v2 file places its sections in a defined order: an optional byte-order mark (BOM), a required H1 heading (# Project or Site Name, the only required section), an optional summary blockquote (> ...), zero or more body sections of any type except headings (such as paragraphs or lists), and zero or more H2 sections (## Section Name) containing Markdown lists of [name](url) links with optional : notes [1] (## Optional remains a convention for secondary links but carries no mechanical semantics in v2 [2]). The spec also proposes that pages agents might need provide clean Markdown versions at the same URL with .md appended (page.html.md) or replacing the extension (page.md), advertised via HTML <link> elements or HTTP Link: headers (rel="alternate" type="text/markdown" for pages and rel="describedby" for the covering llms.txt) [1][2].

Who uses it

Who reads it

Recent coverage

See also

References

  1. Howard, J., The /llms.txt file, v2 (published 2024-09-03; modified 2026-08-10): "We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content. The file can be placed at the site root, or at any path within it, covering the pages under that path. This file offers brief background information, guidance, and links to detailed markdown files. … We furthermore propose that pages with information that agents might need provide a clean markdown version of those pages at the same URL as the original page, either with .md appended (page.html.md) or with the extension replaced by .md (page.md). (URLs without file names should append index.html.md or index.md instead.) To help clients find these files, we recommend using standard link relations: rel="alternate" type="text/markdown" points to the markdown version of a page, and rel="describedby" points to the llms.txt file that covers it. … Agents are expected to view or search llms.txt to find the information they need, then follow the relevant links. … An H1 with the name of the project or site. This is the only required section … While sitemaps list all pages for search engines, llms.txt offers a curated overview for LLMs. … robots.txt lets automated tools know what access to a site is considered acceptable, such as for search indexing bots. llms.txt information is instead used on demand, when an agent needs information about a topic while assisting a user." — https://llmstxt.org
  2. Howard, J., Changes (llmstxt.org, v2, August 2026): "v2 instead states the expectation directly: agents view or search the llms.txt to find what they need, then follow the relevant links, which should point to LLM-friendly content. The context-expansion tooling is no longer part of the proposal, and with it goes the special meaning of the Optional section, which told those tools what to omit. Optional sections are still allowed, and remain a useful convention for secondary links, but they no longer carry mechanical semantics." — https://llmstxt.org/changes.html
  3. Google Search Central, Optimizing your website for generative AI features on Google Search (updated 2026-07-10): "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." — https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
  4. Chrome for Developers, llms.txt (Lighthouse, updated 2026-05-05): "The llms.txt file is an emerging convention used to provide a machine-readable summary of a website's content, specifically designed for LLMs and AI agents. … Lighthouse flags the pages if a server error occurs when attempting to retrieve the llms.txt file. If the file is not provided by the server (resulting in a 404), the audit is marked as Not Applicable (N/A), as providing the file is optional at the moment." — https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt
  5. Rankability, llms.txt Adoption Research Report (2026-09-18): "9.3% of the top 1,000 domains serve a confirmed llms.txt or llms-full.txt file." — https://www.rankability.com/blog/llms-txt-adoption/
  6. Vercel, Make your site readable by AI agents (Knowledge Base, with ora.ai, 2026-08-28): "Of the 329 runs that reached it, 325 had a scorable steering result. Of those 325 runs, 117, or 36%, later fetched a page it listed. Across all 329 reaching runs, 120, or 36%, grounded the final answer in its content." — https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents
  7. Ahrefs, We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read (2026-06-15; data: May 2026): "28% of the 137K domains using Ahrefs Web Analytics publish an llms.txt file. 97% of those files received zero traffic in May 2026." — https://ahrefs.com/blog/llmstxt-study/

JSON · Markdown · OKF v0.2