Crawler logs
The AI crawlers Shruwd recognises, how each hit is verified against vendor IP ranges, and why some are shown but never counted.
Shruwd recognises the AI crawlers below, verifies each hit against the vendor's published IP ranges, and separates fetches made to answer a question from batch crawling for training.
Recognised crawlers
These are the bots Shruwd identifies today, across OpenAI, Anthropic, Perplexity, Apple, ByteDance, Meta, Common Crawl and Amazon:
GPTBot · OAI-SearchBot · ChatGPT-User · ClaudeBot · Claude-User ·
Claude-SearchBot · PerplexityBot · Perplexity-User · Applebot · Bytespider ·
Meta-ExternalAgent · CCBot · Amazonbot
The list is maintained on Shruwd's side and updated as new crawlers appear, so you never need to change anything you deployed. Your brand's Crawlers page lists each bot that has reached you, with its vendor.
Google-Extended and Applebot-Extended are not crawlers
Both are robots.txt tokens that control how your content may be used, and neither ever
appears in a log. Google says Google-Extended "doesn't have a separate HTTP request user
agent string". Apple says Applebot-Extended "does not crawl webpages". The fetching is
done by Googlebot and by Applebot.
So neither has a row on the Crawlers page, and an empty one would tell you nothing. Shruwd
checks Google-Extended where it does exist: the robots.txt diagnostic reports a rule
that disallows it and says what that affects, which is Gemini grounding and AI training,
not AI Overviews. See the findings reference.
Google's AI Overviews and AI Mode are fed by Googlebot, the same crawler as Google Search. There is no separate Google AI crawler to look for in your logs.
Live retrieval versus training
Each bot is classified by what it is for, and the distinction matters more than the totals.
| What it is | Why you care | |
|---|---|---|
| Live retrieval | Fetched to answer a specific person's question, right then | A page it cannot fetch cannot be cited in that answer. This is the number to watch |
| Training | Batch crawling to build a model | Decides what a model learns, not what an answer cites today |
ChatGPT-User and Perplexity-User are live-retrieval crawlers. GPTBot is a training
crawler. Presenting them as one undifferentiated figure would hide the most interesting
signal in the data, so Shruwd never does.
How a hit is verified
A user-agent string is a claim, not evidence — anyone can send GPTBot in a header.
Shruwd checks the client IP of every hit against the ranges the vendor publishes. Those ranges are refreshed every 24 hours. If a refresh fails, the last known-good set is kept rather than marking everything verified or everything unverified — both would quietly corrupt the number.
| Result | What happens |
|---|---|
| Verified | The IP belongs to the vendor. Counted in every metric |
| Unverified | The user-agent claims a bot from an IP the vendor does not own. Shown on the Crawlers page, never counted |
Unverified hits are displayed rather than hidden, because seeing them is useful — but they never reach a headline number. Where a vendor publishes no ranges at all and no reverse-DNS check is documented, its hits stay permanently unverified.
Reading the Crawlers page
Once logs arrive you get:
- Verified hits per day — the timeline. It appears after two days of logs.
- By bot — per-crawler verified hits, unverified hits, and errors. A non-zero error count means that crawler was refused, which is a problem worth a look.
Diagnostics that need logs
| Diagnostic | Needs |
|---|---|
| The server refuses a verified AI crawler | Logs showing the refusal on at least three separate days |
| A page with demand is never crawled | 30 days of uninterrupted logs, and Search Console for the demand signal |
The three-day requirement rules out a transient outage being reported as a block.
The 30-day requirement is a guard, not a delay for its own sake. A brand whose drain broke for a week must not be told a page was never crawled when the truth is that nobody was watching. Search Console is needed alongside it because "never crawled" is only worth raising for a page people actually search for.
Next steps
- AI crawlers — the same list as a public reference, with each vendor's IP-range file
- Choosing a log path — if you have not set this up yet
- How findings work — what these diagnostics produce