llms.txt

How it works

An llms.txt file is a plain markdown file, usually served at /llms.txt on the root of a site. The proposal at llmstxt.org defines a fixed order of sections:

• An H1 with the name of the project or site. This is the only required section.
• A blockquote with a short summary.
• Optional paragraphs or lists with more detail (no headings).
• Optional H2 sections, each holding a list of links in the form [name](url), with a short note after a colon.
• By convention, a section named "Optional" for secondary links an agent can skip when it needs a shorter context.

The idea is that an AI agent reads this small file first, then follows only the links it needs. The proposal also suggests that pages offer a clean markdown copy at the same URL with .md added (for example page.html.md or page.md). The current version, v2, adds link relations so agents can find those copies: rel="alternate" type="text/markdown" for a page's markdown version and rel="describedby" for the llms.txt file that covers it. A file can also sit in a subfolder, such as /docs/llms.txt, and covers only the pages under that path.

The llms-full.txt file is a related but separate convention. It is not defined in the llmstxt.org proposal. Mintlify announced on November 20, 2024 that it generates both /llms.txt and /llms-full.txt for the documentation sites it hosts, describing llms-full.txt as a file that "compiles all your documentation text into a single markdown file."

Where it came from

Jeremy Howard, co-founder of Answer.AI, published the proposal on September 3, 2024. The llmstxt.org page lists a v2 revision dated August 10, 2026, which the author says reflects two years of adoption, mostly in software documentation, where coding agents follow the files to find API references.

The proposal is explicit about how it differs from older root files:

• robots.txt tells automated tools what access is acceptable. In the proposal's words, llms.txt "is instead used on demand, when an agent needs information about a topic while assisting a user." It grants or denies nothing.
• sitemap.xml lists every indexable page. The proposal argues a sitemap is not a substitute because it usually lacks LLM-readable versions, excludes useful external links, and covers more content than fits in a model's context window.

So llms.txt is a curated reading list. It is not an access rule and not a full index.

What AI providers have said

The clearest primary statement comes from Google. Its guide to optimizing for generative AI features says site owners do not need to create "new machine readable files, AI text files, markup, or Markdown to appear in Google Search," and that publishing an llms.txt file "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." Google's documentation updates log dates that clarification to June 15, 2026, and notes that owners may keep the files for other services.

OpenAI's crawler documentation describes how site owners control GPTBot and OAI-SearchBot through robots.txt tags, and treats them as independent settings: you can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot for training. It does not describe llms.txt as an input to ChatGPT search.

OpenAI, Anthropic and Google (for Gemini) publish llms.txt files for their own developer documentation, as the llmstxt.org page points out. That shows the format is useful for documentation consumed by coding tools. It is not a statement that their search or answer products read other sites' files. As of November 2026, no major AI search provider has said it uses llms.txt for ranking or citation.

What the adoption evidence shows

The largest content analysis comes from the Common Crawl Foundation. Its report, What is actually inside llms.txt, examined 584,107 llms.txt documents in the CC-MAIN-2026-30 crawl archive and found that 68.3% were generated from a template rather than written by a person. Wix's default output alone made up 41.34% of the corpus, and 22.56% of files contained no links at all, which removes the point of the format.

That study measured what is inside the files that exist. It is not a count of how many sites publish one. Our reality check on the Common Crawl llms.txt study covers the full numbers, the misuse of llms.txt for crawler directives, and published server logs where the requests for llms.txt came from SEO audit tools rather than answer engines. The short version: widely adopted, largely unvalidated, cheap to do badly.

What to do about it

Publishing a hand-written llms.txt is cheap and reasonable, especially if you run developer documentation that coding agents read. If you do it:

1. Write it yourself. Name the company, summarize what you sell in two sentences, and link to the 10 to 30 pages you would want an assistant to read.
2. Read any file a plugin or CMS created for you. If it has no links or lists every URL on the site, fix it or remove it.
3. Keep every crawler rule in robots.txt. Rules in llms.txt are not enforced.

Then put most of your effort where providers have actually stated requirements:

• Crawl access. Make sure robots.txt, your CDN and your firewall allow the AI crawlers you want, and decide training and search bots separately.
• Server-rendered content. Google says it can process JavaScript that is not blocked, but our write-up of a server-log test of six AI crawlers shows several non-Google crawlers missing pages reachable only through JavaScript links. Serve key text and links in the initial HTML.
• Clear pages. Google's guide says content that people find "unique, compelling, and useful" matters more over time than any other suggestion it makes. A pricing page, product page or comparison page that states facts plainly does more than any index file.

Finally, measure the outcome that matters: whether answer engines mention and cite you on the questions your buyers ask.

Common misconceptions

"llms.txt is a standard." It is a proposal published by one author, open for community input. No standards body has adopted it.

"It is robots.txt for AI." It controls nothing. Crawler access is set in robots.txt and enforced, if at all, at the server or network edge.

"Publishing one improves AI rankings." Google says it ignores the file in Search, and no other major AI search provider has said it uses the file for ranking or citation.