Skip to content
Back to blog

Guide

AI crawlers: three decisions to make, and the block nobody decided on

OpenAI says GPTBot does not decide your place in ChatGPT search. Google says Google-Extended does not touch Google Search. What each AI crawler really controls, a decision table, and the blocks to look for in your CDN and Search Console.

9 min
Illustration: a dotted trail climbing to a mountain pass, blocked halfway up by a rockslide.
Contents
  1. What separates training, search and user-triggered fetching?
  2. Why is Google the odd one out?
  3. What does the shared signal planned for 2027 change?
  4. Which setting fits your situation?
  5. What does a coherent robots.txt look like?
  6. Where do the blocks nobody decided on come from?
  7. How do you check in your logs that the decision holds?
  8. Does an admitted crawler actually read your pages?
  9. Should you publish an llms.txt file?
  10. Where do you start?
  11. Frequently asked questions

Allowing or blocking AI crawlers is not one decision. It is three: letting an engine find you, letting a user have their assistant read your page, and letting a vendor train its models on your content. For a brand that sells, the default position fits in a sentence: open the first two, and treat the third as a rights question, not a visibility one.

The most concrete risk is not a bad call. It is the block nobody decided on: a CDN policy, a Search Console setting, a tag left in an old template. No marketing report flags any of them.

This guide follows each vendor's documentation as read on 5 November 2026. Read it again before you act: on 30 October 2026, Google, Microsoft and Anthropic each announced or dated a change to their controls.

What separates training, search and user-triggered fetching?

Purpose, and therefore what closing them costs you. A training crawler is a bot that collects pages to improve a vendor's future models, with no direct link to a question asked today. A search crawler feeds the index an engine draws answers and links from. A user-triggered fetch starts with a person: the assistant opens a page because it was asked to.

CrawlerFamilyWhat the vendor saysEffect of a block
GPTBotTrainingUsed to make OpenAI's foundation models "more useful and safe"Signals that content should not be used in training
OAI-SearchBotSearchSurfaces websites in ChatGPT's search featuresNot shown in search answers, plain navigational links at most
ChatGPT-UserUser-triggeredCertain user actions, Custom GPTs"robots.txt rules may not apply"
ClaudeBotTrainingContent that could contribute to trainingFuture content excluded from training
Claude-SearchBotSearchImproves search result qualityNo indexing, may reduce visibility and accuracy
Claude-UserUser-triggeredReaches sites when a user asks a questionNo retrieval on request, may reduce visibility
PerplexityBotSearchSurfaces and links websites, not for foundation modelsPerplexity recommends allowing it to appear in its results
Perplexity-UserUser-triggeredUser actions"generally ignores robots.txt rules"
GooglebotSearchGoogle Search and all its featuresPages closed to Search's crawler
Google-ExtendedControl tokenGemini training and grounding in Gemini Apps and Vertex AINo effect on Google Search

Source: OpenAI, Anthropic, Perplexity and Google documentation, read on 5 November 2026.

The table undoes three common assumptions. Closing GPTBot or ClaudeBot does not take you out of search answers: OAI-SearchBot and Claude-SearchBot decide that. Conversely, a rule written against "AI" that also catches those two costs visibility nobody meant to give up. And robots.txt does not reliably govern user-triggered fetches: closing them for certain at OpenAI or Perplexity takes a web application firewall (WAF). OAI-AdsBot is a separate case: it only visits pages submitted as ads on ChatGPT, so it concerns advertisers alone.

Why is Google the odd one out?

Because no separate AI crawler controls AI Overviews or AI Mode. Google says a page qualifies if it is indexed and eligible to show with a snippet, with no additional technical requirement. There are three levers, at three levels:

  • The tag, page by page. nosnippet, data-nosnippet, max-snippet and noindex limit what is shown, across Search in general.
  • The site setting. Search generative AI, in Search Console settings, rolled out worldwide as of 31 August 2026, includes or excludes the site from AI Overviews, AI Mode and generative AI features in Discover. Google says it is not a ranking signal elsewhere in Search and does not affect training.
  • Google-Extended. It never shows up in your logs: Google states it has no separate request user agent. Any line claiming to be it is false.

The trap sits in that last token. It is usually filed under training settings, yet Google also ties it to grounding, the Search index content handed to the model at prompt time, in Gemini Apps and Vertex AI. Closing it is therefore not neutral for your presence in Gemini. For how these features landed in France, see the first AI Overviews figures there.

What does the shared signal planned for 2027 change?

It lets you refuse training in one line, and it calls for two checks. The signal comes from the IETF's AIPREF working group. Microsoft documents its robots.txt token, __usage_ai-training, and Google says it aims to support it in 2027. Anthropic says it has recognized a signal based on a draft from that group since 30 October 2026, rolling out over time, and that a rule naming ClaudeBot takes precedence over the general setting.

  • Microsoft, 1 March 2027. The Bing Blog says the new preferences are honored from 1 February 2027. After 1 March 2027, NOARCHIVE and NOCACHE are no longer read as training or grounding preferences. Grounding is set page by page (ai-grounding:disallow), not in robots.txt, and a robots.txt block on Bingbot still takes priority. Sites that use those tags can record a temporary preference in Bing Webmaster Tools, open from 15 December 2026, before 1 March 2027; otherwise their pages default to allowing grounding until they are recrawled. If those tags sit in your templates, put a date on the work.
  • Google, 2027. If a site refuses training through the shared token without mentioning Google-Extended, Google plans to treat that as disallowing Google-Extended. Google also writes that sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps. To refuse training elsewhere and stay eligible for that grounding, you will have to say so.

Which setting fits your situation?

The one that follows the value of the content, not a stance on AI.

SituationSearchUser-triggeredTraining
Brand that sells: products, prices, offers, public guidesOpenOpenLegal's call, no gain proven either way
Content is the product: press, paid research, databasesDecide by licenseDecide, with a WAF if the refusal must holdClosed
Content you do not fully own: photos, reviews, licensed textOpen, except those pathsSameThose paths closed
Customer area, staging, internal documentsAuthenticationAuthenticationAuthentication

The author's framework, to adapt to your contracts.

The last row matters: RFC 9309 states that these rules "are not a form of access authorization." A robots.txt file is a request addressed to bots, and it publishes the list of paths you would rather not mention.

What does a block cost? Little is known, and nothing about brands. The most serious measurement found is a preprint by Hangcheng Zhao and Ron Berman, revised 15 April 2026: comparing news publishers that block with those that do not, total traffic falls about 7% within six weeks (SimilarWeb, Semrush). The effect on human traffic, from Comscore, is similar in size but imprecise, and the result stops being significant after about 20 weeks. It only shows up for the top 50 sites in the wider sample, and the window ends in May 2024, before AI Overviews. If you see a 23% figure, it comes from the abstract of the first version of that paper, since revised.

As of 5 November 2026, no controlled test was found on what opening or closing training crawlers does for a brand.

What does a coherent robots.txt look like?

A file that names bots by family and repeats its exclusions, because a named group replaces the generic one. An illustrative example, for a fictional brand closing training as a rights choice:

# Search: open
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /account/
Disallow: /cart/

# User-triggered: open
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Disallow: /account/
Disallow: /cart/

# Training: closed
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

# Gemini: training and grounding go together, a separate decision
User-agent: Google-Extended
Allow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /account/
Disallow: /cart/

Four rules to know:

  • No inheritance. Under RFC 9309 a crawler follows the groups that name it, and falls back to * only when none match. A named crawler therefore ignores your generic exclusions.
  • The longest rule wins. The most specific path prevails in a conflict, not the order of the lines.
  • One file per subdomain. Anthropic reminds site owners of this for its crawlers: the blog, the store and the help center are each set on their own.
  • A server error closes everything. If robots.txt returns a 5xx, the standard requires the crawler to assume a complete disallow. On a 4xx, it may read anything. Allow about 24 hours for a change to apply, according to OpenAI and Perplexity.

Where do the blocks nobody decided on come from?

From layers the marketing team never sees. No solid study puts a number on how common they are: only a check of your own stack will tell.

  • The CDN's AI policy. The idea that Cloudflare blocks AI crawlers by default no longer matches its documentation: updated 2 November 2026, it says new domains allow all three families (Search, Agent, Training) and that blocking is a choice. What remains is finding out what was chosen on your side. Cloudflare's Training family includes mixed-purpose crawlers, used for both training and search.
  • Bot protection. Perplexity warns that Cloudflare's Bot Fight Mode cannot be skipped by custom rules and may keep challenging its crawlers. On AWS WAF, the same guide puts the allow rule ahead of the blocking rules.
  • IP blocking. Anthropic warns it may not reliably hold as an opt-out, because it stops its crawlers from reading your robots.txt.
  • Country and rate rules. A France-only filter or a request limit can return an error to a legitimate crawler. Check it on your side: no vendor documents it.
  • Inherited settings. A Search generative AI exclusion in Search Console, nosnippet or NOARCHIVE on a template, a robots.txt copied from staging.

How do you check in your logs that the decision holds?

By looking at who comes, what they receive, and whether it is really them. One command is enough for a first filter, to adapt to your log format:

grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" access.log \
  | awk '{print $1, $9}' | sort | uniq -c | sort -rn | head -30

Four precautions when reading the result:

  • Read CDN logs, not only origin logs. A request refused at the edge never reaches your server: the origin shows an absence, not a refusal.
  • Check the address. A crawler's name can be faked. OpenAI, Anthropic and Perplexity publish their ranges as JSON, and Google documents a reverse DNS check. Perplexity advises checking both together.
  • Be wary of testing with a fake crawler name. A curl from your laptop tests a rule on the name, not how real verified crawlers are treated.
  • Do not conclude from absence. A short window with no visit does not prove a block. Do not panic over 404s either: Vercel measured 34.82% of ChatGPT's fetches and 34.16% of Claude's hitting them, against 8.22% for Googlebot, in December 2024.

The traffic these engines send is a separate topic, covered in the GA4 measurement guide.

Does an admitted crawler actually read your pages?

Not necessarily what a visitor sees. The most solid public measurement found is still Vercel's, published 17 December 2024: none of the major AI crawlers measured executed JavaScript (OpenAI's crawlers, ClaudeBot, Meta-ExternalAgent, Bytespider, PerplexityBot), while Gemini, on Googlebot's infrastructure, and AppleBot rendered it. Copilot was left out. Google, for its part, says it processes JavaScript as long as it is not blocked.

That data is almost two years old, and no recent measurement of similar scale was found as of 5 November 2026. The test costs nothing: take a page's price, warranty or key argument, and look for them in the HTML the server returns, before execution. A page whose proof only exists after rendering is an open door to an empty room, as the homepage audit shows.

Should you publish an llms.txt file?

Not to gain visibility. Google's guide says Google Search does not use it, and that creating one for other services neither helps nor hurts in Search. SE Ranking finds no correlation with citations across nearly 300,000 domains (October 2026). Ahrefs looked at 137,210 domains: of the roughly 38,000 with a valid file, 97% received no requests for it in May 2026, and 96% of the requests that did occur came from bots.

An area of doubt remains: these studies say nothing about future use by other services or agents. Since the cost is low, there is no need to remove a file already in place. Just do not count it as a win, in line with these tactics graded by their evidence.

Where do you start?

  1. Split the decision. Marketing decides search and user-triggered access, legal decides training, engineering executes.
  2. Inventory the four layers. robots.txt on each subdomain, CDN and WAF policy, the Search generative AI setting, template tags.
  3. Write the file by family. Start from the decision table, and set Google-Extended on its own.
  4. Check a week later. Read the edge logs: presence, response codes, addresses.
  5. Date what comes next. Put 1 March 2027 in the calendar if NOARCHIVE or NOCACHE appear on your site, and reread the documentation every quarter.

Step four is the one that gets lost, because it has to come back with every CDN or template change. In Nanga's GEO module, the AI Audit, in beta, gives it the "AI crawler access" section of its Technical tab. The check reads your robots.txt and returns a verdict per AI crawler: Allowed, Blocked, No rule or Unknown. "No rule" describes exactly the crawler reading your site that nobody decided on. The log audit shows which crawlers actually came, how often and on which pages, and compares their addresses with the ranges operators publish; the file is analyzed in your browser and never leaves your machine.

To see what your robots.txt allows, crawler by crawler: request a demo.

Frequently asked questions

What is the difference between GPTBot and OAI-SearchBot?

Purpose. OpenAI's documentation says GPTBot serves model training and OAI-SearchBot surfaces websites in ChatGPT's search features. They are set separately: closing the first does not remove you from search answers, and closing the second does.

Does blocking Google-Extended remove a site from AI Overviews?

No. Google says Google-Extended has no effect on inclusion in Google Search. It governs Gemini training and grounding in Gemini Apps and Vertex AI. For AI Overviews and AI Mode, Google documents its snippet controls and, in Search Console, the Search generative AI setting.

How do you leave AI Overviews and AI Mode without leaving Google Search?

Through Search Console, under Settings, with the Search generative AI control, rolled out worldwide as of 31 August 2026 according to Google. It also covers generative AI features in Discover. It changes neither training nor ranking elsewhere in Search, and you lose the impressions and traffic from those features.

Is robots.txt enough to protect content?

No. The RFC 9309 standard states that these rules are not a form of access authorization, and both OpenAI and Perplexity say their user-triggered fetches may not follow them. A customer area, a staging site or an internal document needs authentication.

Should you create an llms.txt file?

Not to gain visibility. Google says Google Search does not use it, SE Ranking finds no correlation with citations across nearly 300,000 domains, and Ahrefs reports 97% of valid files with no requests in May 2026. Nothing requires removing an existing file.

Sources:

See Nanga in action

Discover how the Marketing Intelligence System turns data into decisions.