Answers covers two kinds of fetch: crawlers that build the search index an assistant answers from, such as OAI-SearchBot and PerplexityBot, and fetchers that read a page for one person's question, such as ChatGPT-User and Perplexity-User. Training is batch crawling to build a model.
The vendors document their own crawlers: OpenAI, Anthropic, Perplexity, Google, Apple.
Training or answers: why the split matters
The two kinds of crawler cost you different things when they are turned away. A training crawler that is refused leaves your pages out of a future model. A crawler that fetches for answers and is refused cannot cite you today. Counting them as one number hides the part worth watching, so Shruwd never does.
It also means a site can keep its content out of training and stay eligible for answers, because the vendors give the two jobs different names. GPTBot collects content that may be used for training, and OAI-SearchBot surfaces sites in ChatGPT's search results. A rule that names the first says nothing to the second.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: / Two caveats from the vendors themselves. Perplexity says Perplexity-User "generally ignores robots.txt rules", because a person asked for the page. And a rule in robots.txt only works on a crawler that reads it: a firewall or CDN setting that refuses AI bots applies whatever the file says.
A user agent is a claim, not evidence
Anyone can send GPTBot in a header, so a log line that says GPTBot proves nothing yet. What proves it is the address the request came from.
- Published IP ranges. Most vendors publish the addresses their crawlers use, as a JSON file. A hit counts when its address is inside them. The table links each vendor's live file. The ranges are not reprinted here, because they change and a stale copy is worse than none.
- Reverse DNS. A few vendors publish no ranges and document a hostname suffix. The address has to resolve to a name under that suffix, and the name has to resolve back to the address.
- Neither. Where a vendor publishes no ranges and documents no hostname, a hit can never be verified, so it should never be trusted as that crawler.
Google-Extended and Applebot-Extended are not crawlers
Both appear in lists of AI crawlers, and neither ever appears in a server log. They are robots.txt tokens that control how content already crawled may be used.
Google-Extended. Google: it "doesn’t have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." It governs Gemini training and grounding, and Google says it does not affect inclusion in Google Search. Google’s crawler documentationApplebot-Extended. Apple: it "does not crawl webpages" and "is only used to determine how to use the data crawled by the Applebot user agent." About Applebot
So blocking Google-Extended does not take a site out of AI Overviews. Google's AI Overviews and AI Mode are fed by Googlebot, the same crawler as Google Search, and there is no separate Google AI crawler to look for in your logs.
How to check if OAI-SearchBot and GPTBot can crawl your site
Check three things, in order: that your robots.txt allows them, that your firewall or CDN lets the real crawlers through, and that they visit. The first takes a minute with the free scan. The other two show in your logs, your server's or your CDN's, matched against the addresses OpenAI publishes. Shruwd reads your logs and does that matching for you, on every plan, including the free one.
- robots.txt. Find the group that names
OAI-SearchBotorGPTBot, orUser-agent: *if neither is named, and look for aDisallowthat covers your pages. The free scan does this on your homepage for OpenAI's, Anthropic's and Perplexity's crawlers and theGoogle-Extendedtoken, with no account. OpenAI says a change can take about a day to reach its crawlers. - Firewall and CDN. Look for a bot-blocking setting or firewall rule in front of the site. Either can turn away a crawler your file lets in. Sending
GPTBotas the user agent from your own machine tests what your server does with that name, not with OpenAI's addresses, so a CDN that checks addresses can treat your test and the real crawler differently. What Cloudflare's AI crawler settings block, and where to see it. - Logs. Take the lines that pass the address check above and read the status code. A 200 means the crawler got the page, and a 403 or a 5xx means it was refused. A refusal made at your CDN shows in the CDN's logs or security analytics, not your server's. Shruwd counts refusals for each crawler.
Frequently asked questions
What is GPTBot?
GPTBot is OpenAI’s crawler for content that may be used to train its models. It is separate from OAI-SearchBot, which surfaces sites in ChatGPT’s search results, and from ChatGPT-User, which fetches a page when a person asks ChatGPT about it.
Does blocking GPTBot remove my site from ChatGPT?
Not by itself. GPTBot is the training crawler, and ChatGPT’s search results come from OAI-SearchBot, so a site that blocks GPTBot and allows OAI-SearchBot stays eligible. A wildcard rule that disallows every crawler blocks both.
Is PerplexityBot a training crawler?
No. Perplexity says PerplexityBot "is designed to surface and link websites in search results on Perplexity" and "is not used to crawl content for AI foundation models". Blocking it takes a site out of the index Perplexity answers from.
Does blocking Google-Extended remove my site from AI Overviews?
No. Google-Extended controls whether content may be used for Gemini training and grounding. AI Overviews and AI Mode follow Googlebot, and Google says Google-Extended does not affect inclusion in Google Search.
How do I see which AI crawlers visit my site?
In your server or CDN logs. Filter on the user agents above, then check each hit’s address against the vendor’s published ranges, because the user agent alone can be faked. Shruwd does both for you, and shows which crawlers your server turned away.
Is there a free tool to check if AI crawlers can reach my site?
For robots.txt, yes: Shruwd’s free scan names the AI crawlers your file blocks from your homepage, without signing up. It can’t tell whether a firewall or CDN refuses a crawler the file allows, because it doesn’t request pages as a crawler. Your server or CDN logs show that, and Shruwd’s free plan reads them, counting a hit only when it comes from the vendor’s own addresses.