# Crawlers tracked

Shruwd recognises the AI crawlers below, verifies each hit against the vendor's published
IP ranges, and separates fetches made to answer a question from batch crawling for
training.

## Recognised crawlers

These are the bots Shruwd identifies today, across OpenAI, Anthropic, Perplexity, Apple,
ByteDance, Meta, Common Crawl and Amazon:

`GPTBot` · `OAI-SearchBot` · `ChatGPT-User` · `ClaudeBot` · `Claude-User` ·
`Claude-SearchBot` · `PerplexityBot` · `Perplexity-User` · `Applebot` · `Bytespider` ·
`Meta-ExternalAgent` · `CCBot` · `Amazonbot`

The list is maintained on Shruwd's side and updated as new crawlers appear, so you never
need to change anything you deployed. Your brand's **Crawlers** page lists each bot that
has reached you, with its vendor.

### Google-Extended and Applebot-Extended are not crawlers

Both are `robots.txt` tokens that control how your content may be used, and neither ever
appears in a log. Google says `Google-Extended` "doesn't have a separate HTTP request user
agent string". Apple says `Applebot-Extended` "does not crawl webpages". The fetching is
done by Googlebot and by Applebot.

So neither has a row on the Crawlers page, and an empty one would tell you nothing. Shruwd
checks `Google-Extended` where it does exist: the `robots.txt` diagnostic reports a rule
that disallows it and says what that affects, which is Gemini grounding and AI training,
not AI Overviews. See the [findings reference](https://shruwd.io/docs/findings/findings-reference.md).

Google's AI Overviews and AI Mode are fed by Googlebot, the same crawler as Google Search.
There is no separate Google AI crawler to look for in your logs.

## Live retrieval versus training

Each bot is classified by what it is for, and the distinction matters more than the
totals.

| | What it is | Why you care |
|---|---|---|
| **Live retrieval** | Fetched to answer a specific person's question, right then | A page it cannot fetch cannot be cited in that answer. This is the number to watch |
| **Training** | Batch crawling to build a model | Decides what a model learns, not what an answer cites today |

`ChatGPT-User` and `Perplexity-User` are live-retrieval crawlers. `GPTBot` is a training
crawler. Presenting them as one undifferentiated figure would hide the most interesting
signal in the data, so Shruwd never does.

## How a hit is verified

A user-agent string is a claim, not evidence — anyone can send `GPTBot` in a header.

Shruwd checks the client IP of every hit against the ranges the vendor publishes. Those
ranges are refreshed every 24 hours. If a refresh fails, the last known-good set is kept
rather than marking everything verified or everything unverified — both would quietly
corrupt the number.

| Result | What happens |
|---|---|
| **Verified** | The IP belongs to the vendor. Counted in every metric |
| **Unverified** | The user-agent claims a bot from an IP the vendor does not own. Shown on the Crawlers page, never counted |

Unverified hits are displayed rather than hidden, because seeing them is useful — but
they never reach a headline number. Where a vendor publishes no ranges at all and no
reverse-DNS check is documented, its hits stay permanently unverified.

## Reading the Crawlers page

Once logs arrive you get:

- **Verified hits per day** — the timeline. It appears after two days of logs.
- **By bot** — per-crawler verified hits, unverified hits, and errors. A non-zero error
  count means that crawler was refused, which is a problem worth a look.

## Diagnostics that need logs

| Diagnostic | Needs |
|---|---|
| The server refuses a verified AI crawler | Logs showing the refusal on at least three separate days |
| A page with demand is never crawled | **30 days** of uninterrupted logs, **and** [Search Console](https://shruwd.io/docs/tracking/connecting-search-console.md) for the demand signal |

The three-day requirement rules out a transient outage being reported as a block.

The 30-day requirement is a guard, not a delay for its own sake. A brand whose drain broke
for a week must not be told a page was never crawled when the truth is that nobody was
watching. Search Console is needed alongside it because "never crawled" is only worth
raising for a page people actually search for.

## Next steps

- [AI crawlers](https://shruwd.io/ai-crawlers) — the same list as a public reference, with each vendor's IP-range file
- [Choosing a log path](https://shruwd.io/docs/crawler-logs/choosing-a-log-path.md) — if you have not set this up yet
- [How findings work](https://shruwd.io/docs/findings/how-findings-work.md) — what these diagnostics produce