Crawler logs
Four ways to send AI-crawler hits to Shruwd. Which one is right depends on whether a CDN answers for your site.
Sending your own server logs tells Shruwd which AI crawlers actually reach your pages and which get refused. It is the only first-party evidence in the product, and it is included on every plan, free one included.
This page picks the right path for your hosting. Each one is a separate guide.
Why bother
Two of the four access diagnostics work from your domain alone. The other two need logs, including the most valuable rule in the catalogue: your server returning an error to a verified AI crawler while serving the same page fine to everyone else. Most people who have this problem do not know it — it is usually a bot-protection setting nobody remembers switching on.
Without logs you still get findings. With logs you get the ones that say we watched this happen, which are the ones worth acting on first.
Pick your path
| Your setup | Path | Code? |
|---|---|---|
| Vercel (Pro or Enterprise), Netlify, Fly and similar | Platform log drain | None |
| Behind Cloudflare, any plan | Forwarder Worker | Deploy a small Worker |
| Cloudflare Enterprise zone | Logpush | None |
| Dynamic origin with no CDN in front | Server middleware | A middleware file |
| Vercel Hobby, no Cloudflare in front | None yet. Hobby has no log drains; put Cloudflare in front, any plan, and use the Worker | — |
When two rows fit, take the higher one. Platform drains and Logpush need no code and cannot drift out of date, and Logpush also records the requests Cloudflare itself refused, which the Worker never sees.
If your site is static, prerendered or ISR, do not use server middleware. Middleware only sees requests that reach your origin, and the pages worth measuring are exactly the ones your CDN answers without ever calling you. On a typical marketing site that hides most of your traffic. Use the Worker instead — it runs before cache lookup.
Client IP is required
Every path must send the client IP of each request.
A user-agent string is a claim — spoofing GPTBot takes seconds. Shruwd verifies each
hit against the crawler vendor's published IP ranges, and only verified hits count toward
anything. A log source with no client IP is refused at setup rather than accepted and
quietly ignored:
text
HTTP 422
No line carried a client IP.
That refusal is deliberate. Accepting the data would mean you did the setup and got a dashboard of numbers that cannot be used.
What happens after logs arrive
Your brand's Crawlers page starts showing hits per day and a per-bot breakdown, split three ways:
| Live retrieval | Verified crawling that feeds AI answers. |
| Training | Batch crawling for model training. |
| Unverified | Claimed to be a crawler and could not be verified. Shown, never counted. |
Each plan includes a monthly line allowance — 100k on Crawl, 500k on Starter, 2M on Pro. Past it, lines are discarded until the month turns and the page says so rather than showing you a flat chart.
Next steps
- Crawlers tracked — which bots are recognised and how they are verified
- Ingest not arriving — if nothing shows up