Crawler logs
Work through the causes of a crawler log drain that stays on awaiting first logs.
Settings → Crawler logs still reads awaiting first logs after you deployed. Work through these in order — the first two catch most cases.
1. Test the endpoint and token directly
This separates configuration from your code entirely, and you can run it from anywhere.
?validate=1 parses your sample and reports on it without writing anything.
curl -X POST "YOUR_ENDPOINT?validate=1" \
-H "X-Shruwd-Ingest-Token: YOUR_TOKEN" \
-H "Content-Type: application/x-ndjson" \
-d '{"timestamp":"2026-01-01T00:00:00Z","host":"example.com","path":"/","user_agent":"GPTBot","status":200,"client_ip":"1.2.3.4"}'
| Response | Means |
|---|---|
usable: true | Endpoint and token are both fine — the problem is in your app or platform. Go to step 2 |
401 | Wrong or revoked token. See below |
404 | Wrong brand in the endpoint URL |
422 | The sample can't be used. reason says why |
Creating a token revokes the previous one. If you minted a new token in Settings after setting up the drain, the one your app is using is dead. Only the newest token works.
2. On Vercel, check the drain
In Team Settings → Drains:
- The drain isn't marked as errored or paused.
- Its sources include
static,lambda,edge,external,redirectandfirewall. - Its environments include
production, and it has no sampling rule.
Then check Team Settings → Security & Privacy → IP Address Visibility is on. Hidden IPs look like a working drain: logs arrive and the card reads active, but every line is refused, so the Crawlers page stays empty.
3. Check the variables in the deployed environment
Set locally and forgotten in the host's configuration is the single most common cause.
Check the values in the environment your site actually runs in, not your local .env.
If you are using server middleware, it logs one
line the first time a send fails — look for [shruwd] log drain not working: in your
app's logs. That line names the cause.
If you switched the middleware to useRuntimeConfig(), confirm both halves: the keys
declared in nuxt.config.ts and the NUXT_ prefix on the variables. Missing either
reads undefined silently.
4. Check whether the request was served from cache
This only applies to server middleware. A request your CDN answers never reaches your origin, so the middleware never sees it.
Check cf-cache-status on the response:
| Value | Reached your origin? |
|---|---|
DYNAMIC or MISS | Yes |
HIT | No — the middleware could not have seen it |
If most of your pages are cached, middleware is the wrong path. Switch to the forwarder Worker or a platform drain.
cf-cache-status: DYNAMIC means Cloudflare did not serve it from Cloudflare's cache. It
does not guarantee the request reached your application — a second CDN behind it can
still answer from static files.
5. Wait a little
Middleware batches: it sends at 20 events or 10 seconds, whichever comes first. One test request will not appear immediately.
Crawler traffic is also genuinely intermittent. A low-traffic site may wait hours between real AI-crawler hits.
Other things worth knowing
Logs arrive but the card shows gaps. Days with data on the Settings card shows how many of the last 30 days received anything. Gaps switch off the diagnostic that needs 30 uninterrupted days.
Logs arrive but nothing is counted. Check the Unverified column on the Crawlers page. Hits from IPs outside the vendor's published ranges are shown but never counted — which is correct, and still proves your pipe works.
It stopped partway through the month. Check Lines this month against your plan's allowance. Past the cap, lines are discarded until the month turns. The Crawlers page says so rather than showing a flat chart.
A 413 response. Your batches are over the size cap. On Cloudflare Logpush, set
max_upload_bytes to 5000000.
Next steps
- Choosing a log path — if you picked the wrong one
- Crawlers tracked — how verification works