Answers covers two kinds of fetch: crawlers that build the search index an assistant answers from, such as OAI-SearchBot and PerplexityBot, and fetchers that read a page for one person's question, such as ChatGPT-User and Perplexity-User. Training is batch crawling to build a model.
The vendors document their own crawlers: OpenAI, Anthropic, Perplexity, Google, Apple.
Training or answers: why the split matters
The two kinds of crawler cost you different things when they are turned away. A training crawler that is refused leaves your pages out of a future model. A crawler that fetches for answers and is refused cannot cite you today. Counting them as one number hides the part worth watching, so Shruwd never does.
It also means a site can keep its content out of training and stay eligible for answers, because the vendors give the two jobs different names. GPTBot collects content that may be used for training, and OAI-SearchBot surfaces sites in ChatGPT's search results. A rule that names the first says nothing to the second.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: / Two caveats from the vendors themselves. Perplexity says Perplexity-User "generally ignores robots.txt rules", because a person asked for the page. And a rule in robots.txt only works on a crawler that reads it: a firewall or CDN setting that refuses AI bots applies whatever the file says.
A user agent is a claim, not evidence
Anyone can send GPTBot in a header, so a log line that says GPTBot proves nothing yet. What proves it is the address the request came from.
- Published IP ranges. Most vendors publish the addresses their crawlers use, as a JSON file. A hit counts when its address is inside them. The table links each vendor's live file. The ranges are not reprinted here, because they change and a stale copy is worse than none.
- Reverse DNS. A few vendors publish no ranges and document a hostname suffix. The address has to resolve to a name under that suffix, and the name has to resolve back to the address.
- Neither. Where a vendor publishes no ranges and documents no hostname, a hit can never be verified, so it should never be trusted as that crawler.
Google-Extended and Applebot-Extended are not crawlers
Both appear in lists of AI crawlers, and neither ever appears in a server log. They are robots.txt tokens that control how content already crawled may be used.
Google-Extended. Google: it "doesn’t have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." It governs Gemini training and grounding, and Google says it does not affect inclusion in Google Search. Google’s crawler documentationApplebot-Extended. Apple: it "does not crawl webpages" and "is only used to determine how to use the data crawled by the Applebot user agent." About Applebot
So blocking Google-Extended does not take a site out of AI Overviews. Google's AI Overviews and AI Mode are fed by Googlebot, the same crawler as Google Search, and there is no separate Google AI crawler to look for in your logs.
Check your own site
The free scan checks your robots.txt for AI crawler rules and tells you which crawlers are blocked, with no account. Whether real crawlers are being refused by a firewall shows in your server logs, or in your CDN's security analytics if the CDN refused them. Shruwd reads your logs for you on every plan, including the free one.
Frequently asked questions
What is GPTBot?
GPTBot is OpenAI’s crawler for content that may be used to train its models. It is separate from OAI-SearchBot, which surfaces sites in ChatGPT’s search results, and from ChatGPT-User, which fetches a page when a person asks ChatGPT about it.
Does blocking GPTBot remove my site from ChatGPT?
Not by itself. GPTBot is the training crawler, and ChatGPT’s search results come from OAI-SearchBot, so a site that blocks GPTBot and allows OAI-SearchBot stays eligible. A wildcard rule that disallows every crawler blocks both.
Is PerplexityBot a training crawler?
No. Perplexity says PerplexityBot "is designed to surface and link websites in search results on Perplexity" and "is not used to crawl content for AI foundation models". Blocking it takes a site out of the index Perplexity answers from.
Does blocking Google-Extended remove my site from AI Overviews?
No. Google-Extended controls whether content may be used for Gemini training and grounding. AI Overviews and AI Mode follow Googlebot, and Google says Google-Extended does not affect inclusion in Google Search.
How do I see which AI crawlers visit my site?
In your server or CDN logs. Filter on the user agents above, then check each hit’s address against the vendor’s published ranges, because the user agent alone can be faked. Shruwd does both for you, and shows which crawlers your server turned away.