Six AI Crawlers Found Zero of 293 Pages. The Links Were JavaScript.
Most advice about AI crawlers and JavaScript is asserted rather than measured. Someone says GPTBot does not render JavaScript, someone else says it probably does now, and the conversation resolves into whichever opinion was stated with more confidence.
On August 19, Search Engine Land published a contributor piece by SEO practitioner Vinicius Stanula built on 41 days of server logs from a site they own. It is a sample of one. It is also a meaningfully higher standard of evidence than this category usually clears, and the result is stark enough to be worth acting on carefully.
The experiment was designed to remove every escape hatch
The property is a Brazilian, Portuguese-language business-classification directory of roughly 2,400 pages, on a fresh domain, with no external links pointing at the pages under test.
Stanula built a 1,062-page test hierarchy across 21 sections:
- 11 sections with hard-coded HTML links, covering 748 pages.
- 10 sections with JavaScript-injected links, covering 293 pages.
Then every fallback was deliberately removed. Sitemaps were made to return 404. Breadcrumbs and hierarchy panels were stripped. The only route into the JavaScript sections was to execute the JavaScript and follow the injected links.
Server-side edge middleware logged bot requests by user agent — roughly 30,180 of them. The split ran 27 days, followed by 14 days after all links were converted to plain HTML.
That design is the reason the result is interesting. It is not measuring whether crawlers can find JavaScript-linked pages when there are five other ways in. It is isolating one variable: will a crawler execute JavaScript to discover a link.
The result
Over the 27-day test phase, GPTBot, ClaudeBot, Bingbot, PerplexityBot, Meta-ExternalAgent and Amazonbot discovered zero of the 293 JavaScript-gated pages.
Not few. None.
Google was the explicit exception, and unevenly so. Google's crawlers executed the JavaScript and followed the injected links, but GoogleOther reached 48% of the JS-linked pages while index-building Googlebot reached only 2%.
There is a quieter finding in the control group that deserves attention: even among pages that were reachable, JavaScript-linked pages took 18% to 27% longer to discover than their HTML equivalents. Rendering is not free even where it happens.
The recovery is the most important part
After the links were converted to plain HTML, GPTBot crawled 250 of the previously unreachable pages in under three hours.
Three hours.
The pages had not been assessed and found wanting. They had not been deemed low quality, thin, or duplicative. They were never evaluated at all, because they were never found.
That distinction is the practical heart of this. A page that is crawled and not cited is a content problem — you are competing and losing, and the fix is editorial. A page that is never crawled is an access problem, and the fix is a template change that can resolve in an afternoon.
Most AI-visibility programs are structured entirely around the first failure mode. Very few check for the second.
Source: Search Engine Land, contributor research by Vinicius Stanula (August 19, 2026)
What this does not establish
The finding is striking, which is exactly when caveats need stating loudly rather than in a footnote.
The denominator is 293, not 1,062 and not 2,400. The zero-discovery result applies to the JavaScript-gated subset only.
The JavaScript condition ran 27 days, not 41. The remaining 14 tested recovery.
The design is deliberately artificial. Most production sites publish a working sitemap — a discovery path this test removed on purpose. Real-world invisibility would almost certainly be far less absolute. If your sitemap is healthy and submitted, your JavaScript-linked pages are probably discoverable by other means.
It is one fresh, unlinked, Portuguese-language directory. Stanula flags the generalization limit directly, noting that established sites with strong signals may see Googlebot invest more in rendering. A mature domain with authority signals is a different case.
Two taxonomy corrections matter. Bingbot is a conventional search crawler that feeds Copilot, not an AI crawler in the same sense as the other five. And because Google's stack did render the JavaScript, the headline "invisible to AI search" does not hold for Google-powered surfaces — AI Overviews, AI Mode and Gemini all sit downstream of crawlers that executed the script.
On self-interest: this is an SEO practitioner testing on their own property and publishing through a Semrush-owned outlet, on a problem they are positioned to advise about. That does not make the logs wrong. It is a reason to want replication.
What survives all of that is narrower than the headline and still actionable: for the non-Google AI crawlers, JavaScript-dependent discovery is at best unreliable, and plain HTML links are free insurance.
The bigger context: the crawl is one-directional
The JavaScript finding lands inside a larger picture that changes how you should read any crawl statistic.
Cloudflare figures cited in IAB Australia's crawler guidance, pulled out by PPC Land on August 2, record that automated requests overtook human ones in web page traffic for the first time on record in mid-2026, reaching 57.5% of requests by June. The majority of what visits the web is no longer a person.
That milestone will get quoted for a year and change nobody's behavior. The number underneath it should:
- Roughly 52% of AI crawler requests feed model training.
- Only about 2.6% are real-time fetches triggered by an actual human question.
For every twenty visits an AI system makes to absorb your content, fewer than one is a live lookup on behalf of somebody who asked a question your page could answer.
This has a direct consequence for how crawl data gets reported internally. A rising AI crawl line is not an engagement metric. It mostly measures how much of your library is being ingested, not how often you are being consulted. Teams presenting crawler volume growth as evidence that AI visibility work is succeeding are reading the wrong signal.
One trend cuts the other way and is genuinely good news: the same guidance records mixed-purpose crawling falling from roughly 49% to 33% of AI requests across the first half of 2026, as operators split do-everything bots into narrower, purpose-declared tokens. That is the precondition for the policy most publishers actually want — decline training, permit live retrieval — which is impossible while one user agent does both.
Source: PPC Land, Bots overtake humans (August 2, 2026)
robots.txt is a request, and one vendor has said so in writing
Every AI crawler control deployed in the past eighteen months rests on the assumption that robots.txt is the contract.
OpenAI's crawler documentation, surfaced by Search Engine Journal's Matt G. Southern on August 14, contains an exception. The wording is that ChatGPT-User visits a page when a ChatGPT user asks a question, and that because those actions are initiated by a user, robots.txt rules may not apply.
The logic mirrors the long-standing exemption for a human clicking a link in a browser. The difference is that the thing doing the clicking now reads the page, summarizes it, and hands the answer to a reader who never arrives.
TollBit's H1 2026 State of the Bots data, cited in the same coverage, puts a footprint on it: roughly 15% of identified AI page-fetchers on European sites reached URLs the site had marked disallowed. Broken out by bot, ChatGPT-User, ByteDance's Bytespider and You.com's Youbot each reached disallowed pages on approximately half of the European sites that listed them.
These are vendor-measured figures from a company selling publisher-side bot monetization, and they deserve that scepticism. The unusual part is that the vendor data and the vendor's own disclosure agree — OpenAI documented the carve-out itself.
A counterintuitive split in the same dataset is worth carrying: Claude-User was disallowed by 9% of European sites versus 26% in North America; Perplexity-User by 13% in Europe versus 26% in North America. Whatever European publishers are doing about AI access, it is happening in courtrooms and licensing negotiations rather than in robots.txt files — which means European exposure is doubled. Fewer sites set a rule at all, and the bot most likely to disregard the rule is the one most likely to have been sent by a paying consumer.
The operational conclusion: stop treating robots.txt as enforcement. Enforcement lives at the edge — WAF rules, network-level bot management, and agent verification that checks the request rather than trusting a user-agent string. And separate policy by bot purpose before writing it, because blocking a training crawler, an agentic fetcher acting for a live user, and a search indexer are three different commercial decisions with three different traffic consequences. Most sites still have one switch for all three.
Source: Search Engine Journal, OpenAI Says Robots.txt May Not Apply To ChatGPT's Fetch Bot (August 14, 2026)
Google's crawlers share one budget
A documentation update on August 3, reported by Search Engine Journal, contains a sentence that changes how crawl budget should be modeled at scale.
Google's crawl budget documentation now recommends serving HTTP 304 Not Modified for pages that have not changed since the last crawl, so Google reuses its cached copy rather than re-downloading the document.
Conditional requests are not new — Googlebot has sent If-Modified-Since and If-None-Match for years. What is new is Google explicitly recommending it in a crawl-budget context, which reframes it from a caching nicety into a lever you are expected to pull. The practical catch is that most large CMS and CDN stacks get this wrong by default, emitting a fresh Last-Modified on every render or a strong ETag that changes when nothing about the content did. Those sites return a full 200 to every conditional request and pay full bandwidth for content Google already has.
The more consequential clarification: all Google crawlers share a single site-level crawl capacity, and high demand from one reduces what is available to the others.
Many technical SEOs have modeled this the other way — image crawling, ordinary Googlebot crawling and product-specific fetchers each on their own allowance. Google is saying it is one pool. If Googlebot-Image is hammering a newly migrated media library, that pressure comes out of the same capacity your product pages need.
Google frames the guidance for sites above roughly a million pages, or above ten thousand rapidly changing ones. The second threshold is lower than people assume and catches sites out: ten thousand frequently changing pages describes a mid-size ecommerce catalog with live pricing, a jobs board, or a publisher with a large syndicated feed — operations that often believe crawl budget is an enterprise concern.
Source: Search Engine Journal, Google crawl budget documentation update (August 3, 2026)
You can now measure the trade for free
Microsoft Clarity added an AI Scrape-to-Referral Ratio to its Bot Analytics report, reported by Barry Schwartz in Search Engine Land on August 13. It divides how many times AI crawlers fetched your pages by how many visits those AI products sent back. The example in Microsoft's own report shows 41 referrals at a ratio of roughly 6,000 scrapes per referral.
Clarity is free, and that is the part that matters.
The scrape-to-referral trade has been the central grievance of the publisher-versus-crawler fight since 2024, and the numbers in circulation came from two kinds of source: Cloudflare's aggregate network data, which describes the web rather than your site, or an individual publisher who ran their own log analysis. Both are contestable. Neither is something a marketing team could produce on a Tuesday afternoon.
Read the 6,000-to-1 figure carefully. It is the illustration in Microsoft's report, not an industry benchmark, and should not be quoted as a market rate. A high ratio also does not automatically mean exploitation — training and indexing crawls are not supposed to produce a referral each. The ratio is most useful as a trend on your own site, and as a comparison across bots: if one agent's fetch volume climbs sharply while its referrals stay flat, that is a specific thing to investigate, and exactly what a single blended number hides.
Source: Search Engine Land, Microsoft Clarity adds AI Scrape-to-Referral Ratio (August 13, 2026)
The enforcement layer does not work either
If robots.txt is a request rather than a rule, the obvious fallback is network-level blocking: identify the crawler by user agent, and when the user agent lies, block by IP address.
A measurement study published on August 11 by Rumaisa Habib and Zakir Durumeric of Stanford, with Sudheesh Singanamalla and Marwan Fayed of Cloudflare, says the second half of that plan does not work. Using an anonymized passive sample of Cloudflare HTTP logs across 195.8 million distinct source addresses in 198 countries, the authors found:
- 5% of client addresses account for 55% of all web requests.
- 44% of client addresses carry at least two distinct user agents.
- One single address produced requests from more than 7,000 user agents.
Only 0.2% of domains in the .com, .net and .org zones map to a unique address, and just 9% of the top million domains have a dedicated one.
Those facts together are the whole argument. Blocking or rate-limiting an address sending you aggressive crawler traffic also blocks whatever else shares it — and the sharing is invisible to you in advance. The authors note the concentration follows ad hoc infrastructure decisions rather than any documented policy, so the parties who absorb the collateral damage cannot be identified beforehand and bear no responsibility for the exposure.
Read the affiliation before the conclusion: one author is at Cloudflare, the dataset is Cloudflare's, and the paper's proposed direction — cryptographic identity — aligns with what a bot-management vendor sells. That does not undermine measurements only a network operator could take. It does mean the recommendation deserves more scepticism than the data.
For a marketing team the practical consequence is narrower than the publisher debate: if your infrastructure colleagues are proposing a blanket IP-level block on AI crawlers, the collateral damage is real, unpredictable, and includes traffic you want.
Source: PPC Land, reporting on the Stanford/Cloudflare measurement study (August 11, 2026)
What blocking actually costs, from someone who did the arithmetic
The publisher argument about AI crawlers has mostly been conducted at the level of principle. On August 4, People Inc. conducted it at the level of arithmetic, which is more useful.
On the company's Q2 earnings call, reported by Digiday, CEO Neil Vogel said People Inc. — formerly Dotdash Meredith — is not turning off Google's crawlers, and gave the numbers behind that decision:
- Google search traffic fell 40% year over year.
- Google search now supplies 21% of the company's traffic, down from 25% a quarter earlier.
- Core sessions fell 22% year over year.
And the half that makes the decision rational: non-session revenue grew 16% year over year to $125 million, rising from 39% to 43% of digital revenue. The business is being rebuilt around revenue that does not require a pageview, and the stated position is that you do not sever a declining channel until the replacement carries the weight.
Twenty-one percent of your traffic is small enough to be losing and large enough to be fatal. That gap is the entire strategy problem, and it is why "just block them" is a slogan rather than a plan.
On the same day, Booking Holdings CEO Glenn Fogel told analysts that Google's AI Overviews put measurable pressure on organic search results during the quarter, while direct traffic remained in the mid-60% range and had not declined. That is corroboration from outside publishing, which matters — the publisher version of this story is easy to dismiss as a business-model complaint.
Source: Digiday, reporting on People Inc.'s Q2 2026 earnings call (August 5, 2026)
A crawl-access checklist
Ordered by cost to fix, cheapest first.
1. Make every important link a real <a href>. If a page's only route in is a click handler or a JavaScript-injected link, at least six major AI crawlers may never reach it. This is usually a template change, and the recovery evidence suggests the payoff is fast.
2. Verify your sitemap actually works. It is the discovery path the experiment deliberately removed, and for most sites it is the thing standing between them and this result. Confirm it returns 200, is current, and is referenced in robots.txt.
3. Pull a per-bot crawl report and look for absences. The failure mode here is silent. You are not looking for errors, you are looking for sections of the site no AI crawler has requested. Nothing in a standard report flags a page that was never asked for.
4. Check your conditional-request handling. If your stack returns 200 with a full body to every If-Modified-Since, you are spending shared crawl capacity to re-serve unchanged pages.
5. Baseline your scrape-to-referral ratio now, before you make any crawler policy change this year. It costs nothing and you will want the before-picture.
6. Move enforcement to the edge, and split policy by purpose. Training crawler, live fetcher, and search indexer are three decisions. One switch for all three is how sites accidentally block the bots that do send readers.
What to watch
Replication of the JavaScript result on an established domain with authority signals and a working sitemap. That is the test that would tell us how much of this applies to a real business site rather than a fresh directory.
Whether non-Google crawlers add rendering. Nothing about this is architecturally permanent; it is a cost decision, and costs fall.
Cloudflare's default changes on September 15, which reclassify how Googlebot is treated under AI-crawler controls. At least one publisher has already reported losing indexing after enabling AI-training blocks. Anyone relying on Cloudflare defaults for crawler policy should be reading the change notes now rather than diagnosing an indexing drop in October.
Whether purpose-declared bot tokens keep expanding. The fall in mixed-purpose crawling from 49% to 33% is the single most useful trend in this space, because it is what makes a nuanced policy possible at all.
The takeaway
There are two ways to be absent from an AI answer. You can be found, evaluated, and beaten — which is a content and authority problem, and a slow one. Or you can never be found at all, which is a plumbing problem, and frequently a fast one.
The industry spends almost all of its attention on the first and almost none on the second. This month produced a logged, 41-day reminder that the second is real, that it is invisible in every standard report, and that when it is fixed the crawlers show up in under three hours.
Check the plumbing before you rewrite the content.
See How AI Describes Your Brand
Discover where you stand in ChatGPT, Gemini, and Perplexity — and how Genmark AI can help you get cited.
Check Your AI Presence