Shruwd
Shruwd

See which AI crawlers really reach your site, and which get turned away.

Shruwd reads your own logs, counts only the crawlers that are who they say they are, and shows you the pages they are refused on. On every plan, the free one included.

Start free See pricing

What the crawlers did, read from your own logs.

Crawler logs are on every plan, including the free one, and the free plan's findings are built on them.

Know which refusals cost you answers.

Your view keeps two kinds of crawler apart: the ones that feed AI answers and the ones that collect training data. Turned away, they cost you different things, so a block that matters never hides inside a total.

It also shows whether your site does what you meant it to. Keeping content out of training can be a deliberate choice. The split tells you whether the crawlers you still want are getting through.

Real crawlers, counted exactly.

A visit counts only once it is confirmed to come from the company it claims. Anything else using a crawler's name is shown on the side and kept out of your numbers and your findings. So Shruwd will not ask you to let in a scraper that borrowed GPTBot's name.

What does count is exact. Your logs are a record of what happened, not a sample, so crawler numbers carry no range. Each visit either happened or it did not.

Free on every plan, and why it can be.

Reading your logs costs Shruwd very little. Tracking AI answers costs money every time a question is asked, and that is what the paid plans pay for. So logs are included on every plan, the free one too, with no time limit.

Each plan includes a monthly allowance of log lines. If you go past it, the page tells you, and counting starts again next month.

Crawler logs by plan
PlanCrawlStarterPro
Crawler logs
Log lines a month100K500K2M
Findings from your logsMost importantAllAll

Only crawler lines count toward the allowance. Your human traffic never does. Compare plans

Questions and answers

Something not covered here? Ask us

Does it work with my host?

If your host can forward its logs, or a CDN you control sits in front of your site, yes. Hosted site builders that offer neither can’t send logs, and everything else in Shruwd works without them.

Will it see crawlers my CDN answers from cache?

Yes, with the setup the guide recommends for your host. It sees pages served from cache as well as the requests that reach your server.

Why is a crawler I recognise marked unverified?

Either the visit did not come from the company it names, or that company publishes no way to check. Either way it is listed and never counted.

Does Shruwd block or allow crawlers for me?

No. It reads your logs and changes nothing on your site or your CDN. When your logs show a crawler being refused, the finding tells you which setting to change.

What happens if my logs stop arriving?

Shruwd tells you in the app, and it does not read the silence as crawlers staying away.

Is this the same as the AI crawler list?

No. The AI crawler reference explains what each crawler does and how a visit is verified. This page is about what Shruwd does with your logs. The AI crawler reference

Find out what the answers say about you.

One measurement, your own crawler data, and the three access findings that matter most. Free, no card, about two minutes.

Start free