JavaScript and SEO: Googlebot and AI crawlers get the same page

The line has been doing the rounds of every deck for two years: "Google runs JavaScript, AI engines don't." Repeated everywhere, measured nowhere, and it sends you hunting for the problem in the wrong place.

On 27 August 2026 we fetched 27 URLs, one per host, eight times each: four User-Agents, a Chrome 126 browser, Googlebot, GPTBot/1.1 and ClaudeBot/1.0, times two full passes five seconds apart. Then a ninth fetch through headless Chrome to get the DOM after JavaScript execution, and a read of robots.txt.

The headline result is not the one we were hoping for, which is exactly what makes it useful. Across the 20 sites where all four User-Agents received a real page, the word gap between Googlebot and GPTBot is zero. All twenty. Nineteen of them serve the exact same word count to a browser, to Googlebot, to GPTBot and to ClaudeBot.

The JavaScript problem is real. It simply is not where people look for it. It plays out on two axes this measurement separates cleanly: access, a 403 against a 200, and rendering, served HTML against executed DOM.

The method, and how to replay it yourself

Only the User-Agent header changes between requests. Accept and Accept-Language are frozen, the order of the four User-Agents is drawn at random on each pass, and an anomaly only counts as differentiated treatment if it repeats on the second pass. That rule alone killed several false positives.

The measurement fits in two terminal lines. The first fetches the page under the identity of your choice and prints the HTTP status with the exact response weight:

curl -s -o page.html -w 'HTTP %{http_code} · %{size_download} bytes' -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' 'https://example.com/'

The second strips comments, scripts, styles and tags from the downloaded file, then counts what is left. The generator below writes both commands for the URL and robot you pick, plus the headless Chrome line that yields the rendered DOM and the robots.txt read.

One checkpoint before going further. On vydera.com/fr, this recipe returns 223,233 bytes, exactly the weight the audit script recorded. The word count, on the other hand, is a deliberate approximation: 1,740 against 1,635 for the stricter script, which only keeps tokens containing at least one letter. Bytes are exact, words are indicative, and what matters is the gap between two measurements, never their absolute value.

First result: the server does not look at who is asking

Across the 20 sites served to all four User-Agents, the only denominator that permits a content comparison:

• 19 serve the exact same word count to all four identities
• 14 return a byte-for-byte identical response
• the word gap between Googlebot and GPTBot is zero on all 20

The five sites identical in words but not in bytes vary something in their response without the content moving. The measurement does not say what, and we will stop at the observation.

Put another way, the overwhelming majority of servers draw no distinction between a browser, Googlebot and an AI robot. The HTML is the same for everyone. Here is the full measurement, filterable: click a site name for the detail on the special cases.

The gap is about access, not content

Five of the 25 browser-reachable sites refuse at least one robot. That is already few, and the honest number is smaller still: only two of those five cases are an AI block in the strict sense.

• nytimes.com and malt.fr: Googlebot gets a 200 and a full page, GPTBot and ClaudeBot a 403. Both sites' robots.txt independently disallows GPTBot. Two corroborated cases, stable across both passes.
• lefigaro.fr and lesechos.fr: all three robots get a 403, Googlebot included, while the browser gets a 200. That is a firewall filtering on the User-Agent header, not an AI policy. Both nevertheless disallow GPTBot in their robots.txt: declaration and enforcement are decided in two different places.
• doctolib.fr: the mirror case. A Googlebot User-Agent gets a 403 while GPTBot sails through. Sent from a consumer IP address, a Googlebot header is not Googlebot, and the real check runs through reverse DNS. That refusal looks far more like spoofing detection than a block on Google, and the opposite must never be written.

The operational lesson fits in one sentence: before concluding a site blocks AI, look at what Googlebot receives in the same run. Of the five refusals observed, three have nothing to do with an AI policy.

Two sites really do serve robots something else

lemonde.fr serves 1.81 times more words to the three robots than to a browser: 11,245 against 6,214, that is 1.95 MB against 1.02 MB. The figure is identical for Googlebot, GPTBot and ClaudeBot, and stable from pass to pass. It is the only site of the 20 comparable ones treating robots differently at the content level.

doctolib.fr goes further: it serves two different pages to two AI robots at the same moment. GPTBot receives 80,892 bytes containing 529 words of editorial content, the "Vivez en meilleure santé" hero. ClaudeBot receives the same 1,640,429 bytes as the browser, whose served text boils down to 228 words of footer. Checked by hand over two extra rounds: it is not the same page, it is a prerendered version reserved for certain robots.

Both cases prove User-Agent differentiation works technically. Neither says anything about its wisdom: serving content that diverges by declared identity is exactly what Google's guidelines call cloaking. Do it knowingly, or not at all.

The real gap: served HTML against executed DOM

This is where the topic becomes a topic. Across the 19 sites already serving full HTML, running JavaScript adds a median factor of only 1.13 to the word count. The median word count at first fetch, across the 25 measured sites, is 1,027. The first-fetch HTML already carries the substance.

Across the four sites the measurement classes as empty shells, under 100 served words in a browser, the gap changes order of magnitude:

• app.diagrams.net: 66 words served, 1,462 once JavaScript runs
• excalidraw.com: 3 words served, 86 after rendering
• todomvc.com, the React demo: 15 words served, 116 after rendering
• squoosh.app: 68 words served, 69 after rendering

None of the four serves more than 70 words. Four cases is too few for a median gain factor: the data file deliberately sets that field to null below five valid measurements, so we cite the cases one by one rather than an average.

One caution on those medians, easy to miss and decisive anyway. The median of served words and the median of rendered words do not cover the same subset of sites: not all of them have a valid render. Their quotient therefore means nothing. The only rendering gain that holds up is computed site by site, then medianised per class, and that is this 1.13.

What this measurement establishes, and precision matters here: the first-fetch HTML is all a fetch delivers. What it does not establish: what GPTBot or ClaudeBot do with that HTML afterwards. We measured HTTP responses, not crawler behaviour. That all four User-Agents receive the same HTML proves the server does not distinguish them, and strictly nothing else.

The awkward part, because it concerns us: vydera.com goes from 1,635 served words to 2,039 after rendering, a factor of 1.25, above the 1.13 median of its own class. digidop.fr, built on the same stack, stays at 1.03. The cause was not measured, and we will not guess at it here.

Three panel sites resist rendering: photopea.com and swile.co gain no words at all, squoosh.app gains exactly one. No cause established there either, so their factor of roughly 1.00 is a floor, not a measurement.

The stack does not predict what gets served

The panel had been sorted in advance into three families: 10 sites expected to serve HTML, Webflow, WordPress and press titles, 10 sites server-rendered then hydrated, Next.js, Nuxt and the like, and 7 expected to be client-rendered. The measurement only confirms the first.

• Served HTML: 10 out of 10. Prediction held, all serve full HTML.
• Server-rendered then hydrated: 7 of the 8 measurable sites. From 527 words for vercel.com to 2,929 for malt.fr. The eighth, doctolib.fr, serves just 228 to a browser.
• Client-rendered: only 4 of 7 are genuinely empty shells. photopea.com serves 430 words, caniuse.com 424, regex101.com 229.

The framework name therefore says nothing about what the server sends. On our panel a Next.js site serves anywhere between 527 and 2,929 words depending on the page and the configuration. The question to put to a headless stack is not is this React, it is is the main content in the HTTP response. Two terminal lines answer it, no architecture meeting required.

The text that is in the response but not in the document

One pattern slips past most audits. Ahrefs' French blog shows 1,174 words of visible text in its served HTML, and carries 137,366 words of French editorial prose inside its script tags. lesechos.fr, same register: 1,345 visible words against 51,758 words of prose in scripts.

That text exists in the HTTP response. It simply is not in the document. What an engine does with it depends on how it reads the response, and the measurement does not settle that: what is measured is where the text sits, not who reads it.

Two cautions on that figure, because it is easy to inflate. Two counts bracket reality without giving it. The broad count picks up code and interface translation dictionaries: on doctolib.fr it returns 160,663 words containing zero article. The prose count, stricter, ignores single-quoted strings and encoded payloads. Only the second is quoted here.

The h1 tags that only exist after rendering

Four of the 25 measured sites carry no h1 at all in the served HTML. Two of them, doctolib.fr and the TodoMVC React demo, have one once JavaScript runs. The other two, lesechos.fr and caniuse.com, have none in either state.

The h1 is the cheapest structural signal there is. If it only appears after execution, it exists for whoever renders the page, not for whoever fetches it. The test takes one line against the file you already downloaded: grep -c -i '<h1' page.html. Same goes for the title, the meta description and the navigation links, all of which should be checked in the raw HTML before anywhere else.

robots.txt: a declaration is not an enforcement

Across the 21 readable robots.txt files in the panel:

• 4 disallow GPTBot, 4 disallow ClaudeBot, 4 disallow Google-Extended, 4 disallow CCBot, 2 disallow PerplexityBot
• none disallows Googlebot
• 15 of the 21 do not even name GPTBot

Not naming a robot means allowing it. Fifteen sites out of twenty-one have therefore made no decision on the subject, which is itself a decision. If you want to settle it, the place for that decision is robots.txt, backed where relevant by an llms.txt file telling AI engines what there is to read on your site.

And the converse holds: lefigaro.fr and lesechos.fr disallow GPTBot in their robots.txt and return a 403 to Googlebot. The two mechanisms are not steered from the same place, are not reviewed together, and contradict each other without anyone noticing.

What this measurement does not prove

• It does not prove GPTBot or ClaudeBot fail to run JavaScript. We measure HTTP responses, not crawlers.
• It says nothing about Google's second rendering pass or its delay. That would require cross-checking server logs against Search Console, which was not done.
• It does not say what the real Googlebot sees. Only the header was spoofed, from a consumer IP address.
• It does not cover firewalls deciding on TLS or HTTP/2 fingerprints: only the User-Agent varied.
• 27 hand-picked sites, one IP, one geography, one moment. No percentage here generalises to the wider web.

Two sites illustrate the difference between "blocking" and "not measured". leboncoin.fr returned a 403 to all four User-Agents, browser included: that is not a blocking site, it is an unmeasured one, and headless Chrome did pull 2,354 words from it. reddit.com left the sample for the mirror reason, its served HTML turning into an anti-bot wall once JavaScript runs.

What to do, in order

1. Measure before believing anything. Two terminal lines give you the HTTP status, the weight and the word count served to each robot. Budget three minutes per page, at no cost.
2. Compare HTTP status codes first, contents second. On our panel, that is where the entirety of the Googlebot versus GPTBot gap plays out.
3. Then compare served HTML against rendered DOM. Past a factor of 1.5 on word count, your main content depends on JavaScript.
4. Check that your h1, your title and your navigation links live in the served HTML. That is the structural minimum, and the cheapest thing to fix.
5. Look at what is asleep inside your script tags. A whole article can sit in the response without being in the document.
6. Decide explicitly in your robots.txt which AI robots you accept, then check your firewall says the same thing. One config line contradicts a policy without warning.

If the page in question is not indexed, rendering is only one cause out of seven, and rarely the right one: we ran all 168 URLs of our own site through Google's inspection API to sort them. On the marked-up side of the topic, see structured data for SEO and AEO. And for what agents come looking for on a site, WebMCP. Definitions: crawl and indexing.