# Choosing a log path

Sending your own server logs tells Shruwd which AI crawlers actually reach your pages
and which get refused. It is the only first-party evidence in the product, and it is
included on every plan, free one included.

This page picks the right path for your hosting. Each one is a separate guide.

## Why bother

Two of the four access diagnostics work from your domain alone. The other two need logs,
including the most valuable rule in the catalogue: your server returning an error to a
verified AI crawler while serving the same page fine to everyone else. Most people who
have this problem do not know it — it is usually a bot-protection setting nobody
remembers switching on.

Without logs you still get findings. With logs you get the ones that say *we watched this
happen*, which are the ones worth acting on first.

## Pick your path

| Your setup | Path | Code? |
|---|---|---|
| Vercel (Pro or Enterprise), Netlify, Fly and similar | [Platform log drain](https://shruwd.io/docs/crawler-logs/platform-log-drains.md) | None |
| Behind Cloudflare, any plan | [Forwarder Worker](https://shruwd.io/docs/crawler-logs/cloudflare-worker.md) | Deploy a small Worker |
| Cloudflare **Enterprise** zone | [Logpush](https://shruwd.io/docs/crawler-logs/cloudflare-logpush.md) | None |
| Dynamic origin with no CDN in front | [Server middleware](https://shruwd.io/docs/crawler-logs/server-middleware.md) | A middleware file |
| Vercel Hobby, no Cloudflare in front | None yet. Hobby has no log drains; put Cloudflare in front, any plan, and use the Worker | — |

When two rows fit, take the higher one. Platform drains and Logpush need no code and
cannot drift out of date, and Logpush also records the requests Cloudflare itself
refused, which the Worker never sees.

> **Warning**
>
> **If your site is static, prerendered or ISR, do not use server middleware.** Middleware
> only sees requests that reach your origin, and the pages worth measuring are exactly the
> ones your CDN answers without ever calling you. On a typical marketing site that hides
> most of your traffic. Use the Worker instead — it runs before cache lookup.

## Client IP is required

Every path must send the client IP of each request.

A user-agent string is a claim — spoofing `GPTBot` takes seconds. Shruwd verifies each
hit against the crawler vendor's published IP ranges, and only verified hits count toward
anything. A log source with no client IP is refused at setup rather than accepted and
quietly ignored:

```
HTTP 422
No line carried a client IP.
```

That refusal is deliberate. Accepting the data would mean you did the setup and got a
dashboard of numbers that cannot be used.

## What happens after logs arrive

Your brand's **Crawlers** page starts showing hits per day and a per-bot breakdown, split
three ways:

| | |
|---|---|
| **Live retrieval** | Verified crawling that feeds AI answers. |
| **Training** | Batch crawling for model training. |
| **Unverified** | Claimed to be a crawler and could not be verified. Shown, never counted. |

Each plan includes a monthly line allowance — 100k on Crawl, 500k on Starter, 2M on Pro.
Past it, lines are discarded until the month turns and the page says so rather than
showing you a flat chart.

## Next steps

- [Crawlers tracked](https://shruwd.io/docs/crawler-logs/crawlers-tracked.md) — which bots are recognised and how they are verified
- [Ingest not arriving](https://shruwd.io/docs/crawler-logs/ingest-not-arriving.md) — if nothing shows up