GPTBot, ClaudeBot, PerplexityBot: who's crawling your site, and should you let them?

You open your logs, or your CDN report, and names nobody introduced you to start showing up: GPTBot, ClaudeBot, OAI-SearchBot, CCBot, Bytespider. At least fourteen user-agents now hit servers on behalf of a language model. The question always lands the same way: what is this, and should I block it?

The topic usually gets settled with opinions. We preferred to go and read what sites actually write. On 27 August 2026 we fetched the robots.txt of 40 domains: 10 French SEO agencies, 10 SEO and GEO tools, 10 French tech media, 9 French e-commerce sites, and vydera.com. 36 files were readable and parseable. In them we looked for 14 AI crawlers, matching the User-agent token exactly.

The result fits in one sentence: the debate has barely started. Eight sites out of 36 name even one AI crawler. The other 28 have never written the word GPTBot.

What the reading shows

• 8 sites out of 36 name at least one AI crawler in their robots.txt.
• 5 sites out of 36 block at least one. All five block CCBot, the Common Crawl robot, and three of them block nothing else.
• 2 sites out of 36 block a training bot belonging to a model vendor: 01net.com and siecledigital.fr.
• 0 sites out of 36 block a live answer engine. OAI-SearchBot, Claude-SearchBot and PerplexityBot are blocked nowhere.
• 0 sites out of 36 block a user-triggered agent, neither ChatGPT-User nor Perplexity-User.
• 0 sites out of 36 set a blanket Disallow: / on User-agent: *. No block in this reading is a side effect of a general shutdown.
• 1 site out of 36 carries a Content-Signal directive.

The denominator is 36, not 40. Three domains return a 403 on their robots.txt: resoneo.com, usine-digitale.fr and laredoute.fr. Those are sites we could not measure, not sites that block, and it is most likely their firewall talking rather than their policy. decathlon.fr makes the point: it only handed over its file on the second attempt, and the audit retries a request solely on a 401, 403 or 429. A fourth domain, 1ere-position.fr, returns a 404: no robots.txt at all, so nothing to parse, but nobody blocked either.

And the figure "5 sites block at least one AI crawler" never travels alone. Out of context it reads as five sites shutting the door on ChatGPT. That is wrong. All five block CCBot, a robot feeding a public corpus reused by third-party training runs, which answers no user question, and three of them block nothing else. Narrow it to the model vendors' own bots and you are down to two sites out of 36. Set aside blogdumoderateur.com, where ccbot is buried in a copied anti-scraper list, and four blocking sites out of 36 remain.

The 14 crawlers, and what blocking one costs you

Before deciding anything, separate four families, because they carry neither the same effect nor the same cost.

• Training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Bytespider, Meta-ExternalAgent. They collect to feed models. Blocking them has no documented effect on whether you appear in answers.
• Live search: OAI-SearchBot, Claude-SearchBot, PerplexityBot. These build the index the engine searches when it needs a source at answer time. Block them and you leave that index.
• User-triggered agents: ChatGPT-User, Perplexity-User. They open a URL because a human just asked for it.
• Generic crawl: CCBot, Amazonbot. They serve no single answer engine, they feed corpora reused elsewhere.

The Named and Blocked columns come from our reading. The Role, If you block it and Verdict columns do not: they come from the vendors' public documentation and from our own judgement. We blocked nothing and measured no citations before or after. That distinction is not methodological vanity: it is the difference between a number and an opinion.

Training versus live search: the call nobody has had to make

This is the sharpest finding of the whole reading, and it shows up as an absence. OpenAI and Anthropic each publish two separate tokens, one for training and one for live search. The split exists precisely so a publisher can say "train elsewhere, but cite me".

Across the 36 files, one site uses that split, and only in one direction. siecledigital.fr blocks GPTBot and ClaudeBot while leaving OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User and Perplexity-User untouched. No site in the panel does the reverse.

More telling still: Claude-SearchBot and Perplexity-User are named by none of the 36 sites. Neither token exists in anyone's robots.txt. That is not a rejection, it is an absence. The decision was never made because the question was never asked.

Six configurations actually observed

numerama.com and frandroid.com: engines in, corpus out

Both write the same thing: User-agent: GPTBot then Allow: /, and the same group for OAI-SearchBot, ChatGPT-User and PerplexityBot. Then User-agent: CCBot followed by Disallow: /. Answer engines get in, the corpus third parties can reuse is closed off. This is exactly the configuration we recommend below, and it exists in production. Two sites, mind you: an existence proof, not a widespread practice.

siecledigital.fr: eight bots closed, five left open

The file sets Disallow: / on GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, Meta-ExternalAgent and Amazonbot. It says nothing about OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User and Perplexity-User, which therefore stay allowed. The effect is the right one: training cut off, citation preserved. The file even carries a section comment reading "BOTS IA - ENTRAÎNEMENT BLOQUÉ", and files Amazonbot elsewhere, under a section headed "BLOCAGE BOTS INDÉSIRABLES". What we describe here is the measurable effect and the text of the file, nothing more.

01net.com: blocking without the engines

CCBot, Bytespider and Meta-ExternalAgent on Disallow: /. No OpenAI, Anthropic or Perplexity bot is named at all. The site wrote a rule about AI crawlers without ever addressing the ones that might cite it.

zdnet.fr and decathlon.fr: slow down rather than shut out

zdnet.fr writes User-agent: CCBot then Crawl-delay: 2. decathlon.fr writes User-Agent: ClaudeBot then Crawl-delay: 5. Neither sets a Disallow. That is a server-load call, not a content call, and decathlon.fr is the only e-commerce site in the panel to name an AI crawler at all.

uplix.fr: the only agency here to mention them, and it opens the door

Six of the fourteen tokens we track get a dedicated group: GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot and Google-Extended. No Disallow anywhere. One detail matters: five of them get Allow: /wp-json/llm-endpoints/, a dedicated path, and only ChatGPT-User gets Allow: /. Three carry a Crawl-delay: 1.

The mechanism, documentary this time, is worth knowing: under the robots.txt standard a bot obeys only the most specific group that names it. Writing a group for GPTBot therefore lifts it out of the User-agent: * rules, even when that group holds no Disallow. An Allow is not a permission, it is an exemption.

blogdumoderateur.com: the block that barely is one

The file does name ccbot, with a disallow: /. Except it does so inside a block of 187 user-agents where Wget, Xenu and site rippers from the 2000s sit side by side. That is not a decision about AI, it is a copied anti-scraper list. Our dataset counts it, because the token is literally there, but flags it listeGenerique and provides a counter that excludes it. That is why "5 sites block" becomes "4" as soon as copied lists come out. Hiding the case would have been simpler, and less honest.

Every block comes from a single category

• 8 French SEO agencies measured: 0 blocks, 1 site names an AI crawler.
• 10 SEO and GEO tools: 0 blocks, 0 sites name one.
• 9 French tech media: 5 sites block, 6 name at least one.
• 8 French e-commerce sites measured: 0 blocks, 1 site names one.

These counts are far too small to talk about rates. Nine media outlets are not the media market, and eight agencies do not stand for the profession. The contrast clearly exists, its magnitude does not survive scrutiny.

What can be said safely: in this panel, the companies whose content is the product are the only ones that wrote a rule. The ones selling visibility, agencies and tools alike, wrote nothing at all. Twenty-eight files out of 36 ignore the question entirely.

Content-Signal: one directive, one site

Content-Signal is a line placed in robots.txt declaring the intended use of the content along three axes: search for classic indexing, ai-input for use inside a generated answer, ai-train for training. It blocks nothing: it states a preference, and no robot is obliged to honour it.

Across the 36 robots.txt files in the panel, exactly one carries such a line, and it is ours: Content-Signal: search=yes, ai-input=yes, ai-train=yes.

That is nothing to be smug about. Vydera is also one of the 28 sites naming no AI crawler at all. Our robots.txt runs to three lines: we declare a preference without naming a single bot, and we wrote it before running this audit. A directive nobody else writes and nobody is bound to follow remains a weak signal.

The configuration we recommend, and its builder

The right question is not "AI, yes or no". It is "on which axis".

• Live search: let them in. This is the only group whose blocking carries a direct, documented cost to your visibility. Leaving the index of OAI-SearchBot, Claude-SearchBot or PerplexityBot means giving up on being cited in their answers.
• User-triggered agents: let them in. Behind ChatGPT-User or Perplexity-User sits a human who just asked for your page. Blocking here amounts to shutting the door on a visitor.
• Training: your call. If your content is your product, close it. If your content is your marketing, the question carries far less weight than it appears to.
• Generic crawl: your call, but less symmetric. CCBot feeds a public corpus reused by third parties you do not get to pick. It is the one bot that all five blocking sites in the panel close, without exception.
• anthropic-ai: leave it alone. A legacy token, superseded by ClaudeBot. Keeping it costs nothing, blocking it gains nothing.

Three mistakes to avoid, in order of damage:

1. Targeting AI through the User-agent: * group. A blanket Disallow: / shuts out Googlebot too. None of the 36 sites in the panel does this, which is good news.
2. Believing robots.txt erases the past. A directive applies to future crawls. It withdraws nothing already collected.
3. Confusing robots.txt with noindex. The first blocks crawling, it does not stop a URL discovered through a link from surfacing elsewhere. The second acts on indexing. They are not two settings of the same dial.

Running the reading on your own site

The method fits in a hundred lines of Node, with no dependencies and no paid tool: a GET on https://domain/robots.txt, redirects followed, a 20-second timeout, a www fallback, and a second attempt with a named audit user-agent on 401, 403 or 429. Then a split into groups and an exact token comparison.

Three counting decisions change the result completely, and deserve to be written down:

• A Disallow inherited from User-agent: * never counts as an AI block. It is not a decision taken about those bots.
• A bot named inside a block of more than twenty user-agents is flagged as a copied list, and a second counter excludes it.
• A 404 is not a 403. No file means nobody is blocked. A 403 means we could not measure, and the domain leaves the denominator.

For a single site, a curl -s https://your-domain.com/robots.txt and a search for the fourteen tokens will do. It is the fastest check in the whole technical audit, and it still comes after internal linking and tags far too often.

What this reading does not say

Worth stating plainly, because this is where most articles on the subject go off the rails.

• It measures a declared intention, never compliance. Nothing here lets anyone claim a bot honoured or ignored a directive. That takes server logs and a cross-check against the address ranges vendors publish.
• It does not measure the intent behind a block. In a text file, a deliberate editorial choice and a list copied from a 2023 forum look remarkably alike.
• It does not measure the effect of a block on visibility. No before-and-after test, no citation measurement. The link between blocking OAI-SearchBot and vanishing from ChatGPT answers comes from OpenAI's documentation.
• It says nothing about who crawls vydera.com. The site runs on Webflow, which does not expose server logs. No hit volume, no crawl frequency appears in this article, and that is deliberate.
• The panel is not representative of the French web. It is hand-built across four chosen sectors: it represents the four worlds where the question matters to our readers.
• The reading is dated. Media robots.txt files change fast, sometimes several times a quarter. This figure holds for 27 August 2026 and should be read with its date attached.

That leaves the decision, and it is simpler than it looks. Let in what cites you, weigh what trains on you, and check that your file actually says what you think it says. To go further on how engines pick their sources, see our llms.txt guide, our method to optimise content for AI citations, and the query fan-out mechanism that decides which queries actually get run.