Common Crawl Read 584,107 llms.txt Files. 68% Were Template Output.
For two years, llms.txt has been the first item on nearly every GEO checklist. Put a curated Markdown file at the root of your domain, the argument goes, and language models will find your best pages faster and cite you more often. It takes an afternoon. It has a spec. It feels like something a technical SEO should have done last quarter.
Almost nobody checked what was in the files.
The Common Crawl Foundation has now done it. Malte Ostendorff, a senior research engineer at the foundation, analyzed 584,107 non-empty llms.txt files found in the CC-MAIN-2026-30 archive, crawled between July 7 and July 25, 2026 across 2.14 billion pages. The findings were published on August 31, 2026 with an interactive report, and Search Engine Journal covered them on September 8.
68.27% of the files were generated by a plugin or a template rather than written. 22.56% contained no links at all.
That second number is the one worth sitting with. A curated list of URLs is the entire proposal of the format. More than a fifth of the files implementing it contain no URLs.
This post is not an argument that llms.txt is harmful, or that it never helps — there is no controlled study establishing either. It is an argument about a failure mode that keeps recurring in AI-visibility work: a tactic can be cheap, widely adopted, endorsed by everyone in your feed, and still sit on almost no evidence. The response to that is not cynicism. It is knowing exactly what you have and what it is doing.
What the study actually counted
The headline figures, with their denominators attached.
Of 584,107 non-empty llms.txt files in a single Common Crawl archive:
- 68.27% were plugin or template output rather than authored text.
- Wix's default output alone accounts for 41.34% of every llms.txt file in the archive. One CMS default is four in every ten files Common Crawl could read.
- 22.56% contained no links whatsoever.
- Roughly 6% carry rules nothing enforces — rate limits, copyright notices, usage terms addressed to nobody in particular.
- All in One SEO installations produce a median of 138 links. That is a different failure than an empty file and arguably a worse one: a 138-link index generated from site structure is not curation, it is a sitemap with a different file extension.
Take those together and the shape of the population is clear: this is overwhelmingly a file that software wrote, for a reason nobody at the site can articulate, because a plugin update added it.
One caution on arithmetic, because it applies to much of the commentary circulating about this study. The report gives the templated share and the linkless share separately. It does not publish the cross-tab. You cannot multiply 68.27% by 22.56% to get the share that is both machine-generated and empty, and you cannot subtract your way to "the share of files that are hand-written and useful." The published data does not answer those questions, and any figure combining the two is somebody's arithmetic, not Common Crawl's finding.
The 1,570 files that thought they were robots.txt
The most instructive finding in the report is also the smallest.
1,570 files, or 0.27%, wrote crawler directives into llms.txt — rate limits, disallow-style rules, denials aimed at named bots. Thirty-two of them tried specifically to block Common Crawl's own crawler, CCBot, that way.
None of them blocked it. Access control is read from robots.txt. llms.txt is not an access control file, has never been one, and no crawler operator has ever claimed to enforce directives found in it.
So thirty-two site owners believe they have opted out of the largest public web archive, and they have not. Their pages were crawled, counted, and then reported in a study about the file they used to try to stop it.
The people who wrote those directives were not careless. They were reasoning by analogy from a file that looks like robots.txt, sits where robots.txt sits, and is described in the same breath in most GEO explainers. The format's resemblance to an enforcement mechanism is doing work its actual status does not support.
If you have crawler rules in your llms.txt, they are decoration. Move them to robots.txt today. That is the single most concretely actionable sentence in this entire study, and it costs nothing.
This is not an adoption statistic, and it will be quoted as one
Read the study for what it measured. Common Crawl examined what is inside the files it found. It did not measure what share of websites publish one.
That distinction will not survive contact with LinkedIn. "68% of llms.txt files are template output" becomes "68% of sites do llms.txt wrong," becomes "most sites' AI visibility setup is broken," becomes a slide in somebody's deck. Each step is a small deformation and the end state is a claim the data never made.
The adoption question has its own, much weaker evidence: estimates range from about 5.61% of the top 10,000 sites in June 2026 up to roughly 10% in vendor guides. Neither is what Common Crawl counted, and they disagree by nearly a factor of two.
Common Crawl is unusually explicit about its own limits. The study is one crawl snapshot and cannot show a trend. The population is limited to hosts it successfully fetched with a parseable text response; a further 519,484 HTML responses may be hiding well-formed files behind misconfigured servers, so the denominator is not every llms.txt file on the web. Coverage skews toward English-language and larger hosts, and the abuse counts are described as lower bounds from lexical matching, not classifier output.
One more thing separates this from most measurement in the AI-visibility space. Common Crawl is a nonprofit that sells nothing related to llms.txt, and its most awkward finding — that sites trying to block its own crawler failed — cuts against its convenience rather than for it. In a field where nearly every load-bearing number is produced by a company selling the instrument that produced it, a disclosed archive, a named sample, and a finding that complicates the author's position is worth more than the precision of any single percentage in the report.
Source: Common Crawl Foundation, What is actually inside llms.txt (published August 31, 2026)
The consumers the format was built for are largely not consuming it
The Common Crawl study tells you what publishers produced. A separate strand of evidence tells you who is asking for it, and the answer is uncomfortable.
In an r/TechSEO thread on August 29, 2026, someone asked whether anyone had had success serving Markdown files for AI crawlers. John Mueller, Google Search Advocate, answered from his own logs: the only crawlers sending an Accept header claiming to accept Markdown on his test sites were SEO tools. He added "ymmv," noted that his server setups do not log the Accept header by default, and advised site owners to start logging it and check their own metrics before generating Markdown for bots. Search Engine Journal reported the exchange on August 31.
Two things about that before it gets over-read. It is a personal Reddit comment, not a Google announcement or a Search Central post, and it carries no sample size, site count, traffic volume or observation window. Mueller flagged the measurement problem himself in the same breath he reported the result.
There is also a mechanism distinction here that decides whether the remark means anything for your site, and it is exactly the detail most likely to be flattened in summary. Mueller measured content negotiation — the Accept: text/markdown request header, a client declaring which formats it will take, with the server choosing which representation to return at the same URL. He did not measure dedicated Markdown URLs, the other mechanism, where you publish /page and /page.md and a crawler asks for the Markdown by name. His observation says nothing about that path.
On that second path there is contradicting data, and it matters. Dries Buytaert published an analysis on March 5, 2026 covering a month of Cloudflare logs for a single personal site in January 2026. It agrees with Mueller on the narrow point — no AI crawler used content negotiation, and llms.txt requests came only from SEO audit tools — but it found GPTBot requesting the .md version in 34.8% of its requests via dedicated URLs, and roughly 7% more total crawler traffic as a result.
So the accurate summary is two-sided: no AI crawler negotiated for Markdown, and one of them asked for the file by name a third of the time. "AI crawlers don't want Markdown" is not what either dataset supports. Both samples are also very small — Mueller's personal test sites, Buytaert's single site over one month — and neither should be generalized to a large commercial domain with different crawler exposure.
What both agree on independently is the finding relevant here: the llms.txt requests in their logs came from SEO audit tools, not from the answer engines the format exists to serve. The format was proposed so language models could find a site's best pages. On the only logs anyone has published, the things fetching it are the tools that audit whether you have one.
Sources: Search Engine Journal, Google's Mueller Shares Their Experience With Markdown For AI SEO, by Roger Montti (August 31, 2026) · Dries Buytaert, analysis of AI crawler behavior from one month of Cloudflare logs (March 5, 2026)
The number that should actually decide this
Buried in Buytaert's analysis is the figure that ought to govern every decision in this category, and almost nobody quotes it. On his site, roughly 1,241 pages were crawled per citation.
That ratio is the reason to be skeptical of visibility gains from making yourself cheaper to crawl. Crawl volume is abundant. Citation is scarce. Cutting your page's byte cost, or handing a crawler a tidier index of your URLs, operates on the abundant side. It does not obviously touch the scarce side. That is the strongest honest argument against llms.txt, Markdown mirrors, and every other "make the machine's job easier" tactic — and it is an argument about expected return, not about harm. The files are not hurting anything. They are competing for engineering time against work on the scarce side.
One caveat, because the argument is not airtight: this is a ratio from one personal site over one month, not an industry benchmark. Treat it as the right shape of question to ask about your own logs, not as a constant.
A very large publisher is doing this anyway, and cannot show you a number
If the evidence is this thin, why is serious money going into it?
In a Digiday piece published on August 26, 2026, USA TODAY Co. — the renamed Gannett, trading as NYSE: TDAY — described a program to rewrite how its pages are structured so AI crawlers can parse them. Kara Chiles, senior vice president of product management, said the work includes testing conversions of webpages into Markdown alongside changes to templates and metadata, with the goal of making the company's reporting easier for AI systems to access, understand and cite. Chiles was blunt about what readers will notice: "A lot of what we're doing right now is probably not going to be recognizable to the human end user."
The framing comes from the top. On the company's second-quarter 2026 earnings call on August 6, chairman and chief executive Mike Reed told investors, "We recognize that we have to create and format content for humans and for machines." The stated commercial goal is licensing.
Here is the part that reframes the whole exercise. The same company told Digiday it blocks roughly 99% of self-identified AI bots by default and whitelists only approved partners. That figure is a company self-report with no stated methodology, measurement window, or definition of "self-identified."
A publisher that blocks 99% of AI bots is not optimizing to be crawled. It is optimizing to be a better-formatted asset inside a contract. It is formatting for a named counterparty, not for the open crawler population that llms.txt addresses — a fundamentally different economic activity from what a mid-market brand is doing when it adds a Markdown index to its root directory, and not validation of it.
And the underlying premise remains contested by the largest analysis anyone has published. Promptwatch, a GEO platform vendor, examined more than 1.6 million citations in AI answers and found Markdown did not increase the likelihood of citation. The figure running the other way — roughly a 90% reduction in tokens from serving Markdown — comes from TollBit, whose chief executive Toshit Panigrahi cites it and whose company sells bot-access monetization infrastructure and benefits directly from publishers adopting the approach. Both sides of that argument are being made by companies that sell into it. And a token-count saving is a cost fact about the model's side of the transaction. It is not a citation fact about yours.
USA TODAY Co. published no citation rate, no impression count, and no referral traffic figure. It said it is monitoring which of its content gets cited in Google AI Overviews and tracking AI impressions, and released none of it. Improving citation rates is a stated objective there, not a measured outcome, and Markdown is one of several approaches being tested rather than a completed conversion. A large publisher has concluded the machine-readable version of its site is a separate product from the human one and staffed it accordingly — and even with that budget, it cannot yet show you a number proving the bet.
What the evidence does support
None of the above says do nothing. It says do the things that have logs behind them. Ordered cheapest first.
1. Log the Accept header, then look. This is Mueller's own advice and it is nearly free. Most default server configurations do not log it, which means most site owners have no idea whether anything is negotiating for Markdown on their domain and are reasoning entirely from other people's anecdotes about other people's sites. An afternoon of log configuration replaces a year of argument.
2. Move every crawler directive out of llms.txt and into robots.txt. Per Common Crawl, 1,570 sites have rules sitting in a file that enforces nothing. If you inherited an llms.txt from a plugin, read it — you may be one of them.
3. If you already serve .md URLs, check whether named crawlers request them. That is the mechanism with log evidence behind it: GPTBot at 34.8% of its requests on Buytaert's site. If you are deciding whether to build it at all, the crawl-to-citation ratio is the number that should decide, not the header.
4. Audit the file you already have. If yours was produced by a plugin, it is statistically likely to be indistinguishable from hundreds of thousands of others, and if it is one of the 22.56% with no links, it is not implementing the format at all. Either fix it into a genuinely curated list of your best pages or accept that it is inert. Both are defensible. Believing it works when it contains nothing is not.
5. Stop treating training opt-outs as a ranking risk. Apple's "About Applebot" support page, published September 4, 2026 and reported by Barry Schwartz at Search Engine Roundtable on September 7, now carries one additional sentence: "Site rules for Applebot-Extended are not considered in ranking for Search." Applebot-Extended is not a crawler — Apple's own documentation says it "does not crawl webpages" — it is a signal governing whether already-crawled content may train Apple's generative models. The page already said pages disallowing it could still appear in Apple Search results; the new line extends the assurance from inclusion to ranking. This is an unverifiable first-party statement about a proprietary system, and Apple is the only party that can check it. But if you were withholding a training opt-out on the theory that it carried a quiet ranking cost at Apple, the documentation now contradicts the theory, and the documentation is all the evidence there is.
6. Spend the recovered time on third-party presence. Several vendor datasets converge, from different directions, on brand-owned pages being a small minority of what AI answers cite on consumer product queries. Shero Commerce, a Shopify agency that sells AI SEO services, collected 1,851 cited sources across 60 product categories in July 2026 and found 2.8% were brand-owned pages. Azoma, an agentic-commerce vendor, put brand-owned sites at 3% of ChatGPT product citations in its Q2 2026 tracking, and Profound has separately reported roughly 4% across about 27 million AI citations. Every one of those producers sells the remedy their finding implies — and a 2025 Yext analysis of 6.8 million citations reported 86% from "brand-managed" sources, using a broader definition that includes listings and profiles a brand controls without owning the domain. The definitional gap is itself the finding. What survives it: on consumer product queries, third-party editorial carries a large share of the load. If you sell B2B software or services, none of these numbers describe your surface and nobody has published one that does.
If you want one instrument rather than six, the thing worth measuring is not whether your llms.txt is well-formed. It is whether the answer engines mention you at all, on the questions your buyers actually ask — which is what Genmark AI exists to measure, and the only feedback loop that will tell you whether any of this worked.
Sources: Search Engine Roundtable, Site Rules For Applebot-Extended Are Not Considered In Ranking For Search, by Barry Schwartz (September 7, 2026) · Search Engine Journal, AI Tools Recommend Brands But Cite Other Sites, Data Shows (August 26, 2026)
The gates that sit above all of this
There is a structural reason the llms.txt debate feels larger than its effect size, and it is worth naming.
Page-level optimization now sits downstream of two decisions you cannot inspect.
Gate one: did the assistant's retrieval layer license an index containing your page at all? Independent index vendors are a real market — Keenable came out of stealth on August 25, 2026 with $26 million from Accel and an unaudited claim to more than 100 billion documents, sold wholesale to AI labs and agent builders. Gate two: if you are a source rather than a destination, did the publisher permit that crawler? USA TODAY Co.'s 99% block rate is one answer.
Neither gate is published, and there is no rank check for either. A page can be crawled by Googlebot, crawled by OAI-SearchBot, rank respectably in both, and still be invisible to an agent whose framework calls a third-party API that never saw it. llms.txt sits below both gates, and expecting it to carry the weight the checklists assign it is a category error.
Source: TechCrunch, Accel-backed Keenable is indexing the web for AI agents (August 25, 2026)
Caveats, stated properly
Everything above rests on evidence with real limits, and the limits should travel with the claims.
What the Common Crawl sample is not. One snapshot, no trend. Limited to hosts fetched successfully with a parseable text response, with 519,484 HTML responses potentially hiding well-formed files behind misconfigured servers. Skewed toward English-language and larger hosts. Abuse counts are lower bounds from lexical matching, not classifier output. And it is a content analysis, not an adoption census.
What nobody has established. There is no controlled experiment testing whether publishing an llms.txt file changes citation rates — not a positive one, not a negative one. Nothing here shows llms.txt is useless. It shows that most files in one large sample are boilerplate, that the llms.txt requests in two independent log studies came from SEO tools, and that the largest published citation analysis found no Markdown effect. Absence of demonstrated benefit is not demonstrated absence of benefit, and anyone telling you otherwise is over-reading the same data I am. The defensible verdict: widely adopted, largely unvalidated, cheap to do badly.
Vendor self-interest runs through most of this. Promptwatch sells a GEO platform. TollBit sells bot-access monetization. Shero Commerce, Azoma, Profound and Yext all sell remedies their findings imply. The two least-conflicted sources in this piece are Common Crawl, a nonprofit with nothing to sell on this question, and Mueller's personal logs, which come with no sample size at all. That is the evidence base, and it is thinner than the confidence in the discourse.
And our own position, since it is fair to ask. genmark.ai publishes an llms.txt. It is hand-written rather than plugin-generated, and it carries roughly twenty curated links to the pages we would want an engine to read. We have no evidence it has produced a single citation, and we have not run a test capable of telling us. We keep it because it costs an hour a quarter and because a hand-written one is at least an honest artifact. That is the whole justification — not a recommendation dressed up as a measurement, and you should treat anyone else's llms.txt advice the same way until somebody publishes a controlled result.
The takeaway
llms.txt is the most-adopted tactic in AI visibility with the least evidence behind it. That is not a scandal — it is what an immature discipline looks like when a plausible idea gets a plugin before it gets a study.
The practical stance is narrow. A hand-written llms.txt is cheap and probably harmless. A plugin-generated one with no links is cheaper still and definitely inert. Crawler rules inside either do nothing, and if you have them, you are one of 1,570 sites with a security control that is actually a comment. The hours you would spend perfecting the file are better spent on the scarce side of the crawl-to-citation ratio.
Most of all, notice the shape of this story, because it will recur with the next format. A tactic spread for two years on a promise nobody measured. The first serious measurement came from the one organization with nothing to sell. And its most useful finding was not that the tactic fails — it was that most of the people doing it had no idea what was in their own file.
Go read yours.
See How AI Describes Your Brand
Discover where you stand in ChatGPT, Gemini, and Perplexity — and how Genmark AI can help you get cited.
Check Your AI Presence