Platform log drains

If your host can forward request logs to an HTTP endpoint — Vercel, Netlify, Fly and most others — this is the whole setup. No code, and the logs are edge logs, so they include requests your CDN answered without touching your origin.

Prerequisites

• Permissions: admin or owner.
• A host with an HTTP log drain feature. These are often on a paid tier; check your platform's current pricing.

1. Get your endpoint and token

1. Open the brand's Settings → Crawler logs.
2. Select Create token.
3. Copy the endpoint and the token.

2. Add the drain on your platform

Point your platform's log drain at the endpoint with the token as a header.

On Vercel

Drains need a Pro or Enterprise team. On Hobby with Cloudflare in front of the site, use the forwarder Worker instead. Hobby without Cloudflare in front has no log path yet.

1. Open Team Settings → Drains → Add Drain and choose Logs.
2. Name the drain and select the project.
3. Select the sources static, lambda, edge, external, redirect and firewall. firewall is the one that records the requests Vercel's firewall refused.
4. Select the production environment. Add no sampling rule, so every request is sent.
5. Choose Custom Endpoint: the endpoint from step 1, format NDJSON or JSON, and the custom header X-Shruwd-Ingest-Token: YOUR_TOKEN.
6. Select Create Drain. Vercel tests the endpoint, which answers 200.

3. Check the fields

Shruwd reads these fields per line. Common alternative spellings — ClientIP, clientip, remote_addr — are understood, so most platforms work as they come. Vercel's format works as it is, including the request details it nests under proxy.

If your host offers drain sampling, keep it at 100%. Crawler numbers are presented as exact counts, and a sampled drain would make them partial.

A single line looks like this:

{"timestamp":"2026-09-09T10:00:00Z","host":"example.com","path":"/pricing","user_agent":"Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)","status":200,"client_ip":"23.98.142.176","method":"GET","bytes":18234}

4. Test before relying on it

Add ?validate=1 to the endpoint to parse a sample and report on it without writing anything.

curl -X POST "YOUR_ENDPOINT?validate=1" \
  -H "X-Shruwd-Ingest-Token: YOUR_TOKEN" \
  -H "Content-Type: application/x-ndjson" \
  -d '{"timestamp":"2026-01-01T00:00:00Z","host":"example.com","path":"/","user_agent":"GPTBot","status":200,"client_ip":"1.2.3.4"}'

Invoke-RestMethod -Method Post `
  -Uri "YOUR_ENDPOINT?validate=1" `
  -Headers @{ "X-Shruwd-Ingest-Token" = "YOUR_TOKEN" } `
  -Body '{"timestamp":"2026-01-01T00:00:00Z","host":"example.com","path":"/","user_agent":"GPTBot","status":200,"client_ip":"1.2.3.4"}' `
  -ContentType "application/x-ndjson"

usable: true means the endpoint and token are both good. Anything else names what is missing.

5. Confirm it is live

Within a few minutes of real traffic, Settings → Crawler logs moves from awaiting first logs to active, and Last received fills in.

Response codes

Next steps

• Ingest not arriving — if the card stays on awaiting first logs
• Crawlers tracked — what gets recognised