# Platform log drains

If your host can forward request logs to an HTTP endpoint — Vercel, Netlify, Fly and
most others — this is the whole setup. No code, and the logs are edge logs, so they
include requests your CDN answered without touching your origin.

## Prerequisites

- **Permissions:** admin or owner.
- A host with an HTTP log drain feature. These are often on a paid tier; check your
  platform's current pricing.

## 1. Get your endpoint and token

1. Open the brand's **Settings → Crawler logs**.
2. Select **Create token**.
3. Copy the **endpoint** and the **token**.

> **Warning**
>
> The token is shown **once**. It cannot be retrieved later, only replaced — and creating a
> new one immediately revokes the old one, which will stop a drain that is already working.

## 2. Add the drain on your platform

Point your platform's log drain at the endpoint with the token as a header.

| Setting | Value |
|---|---|
| URL | the **endpoint** from step 1 |
| Header | `X-Shruwd-Ingest-Token: YOUR_TOKEN` |
| Content type | `application/x-ndjson` |
| Format | NDJSON, one JSON object per line, or one JSON array of objects |

### On Vercel

Drains need a Pro or Enterprise team. On Hobby with Cloudflare in front of the site, use
the [forwarder Worker](https://shruwd.io/docs/crawler-logs/cloudflare-worker.md) instead. Hobby without
Cloudflare in front has no log path yet.

1. Open **Team Settings → Drains → Add Drain** and choose **Logs**.
2. Name the drain and select the project.
3. Select the sources `static`, `lambda`, `edge`, `external`, `redirect` and `firewall`.
   `firewall` is the one that records the requests Vercel's firewall refused.
4. Select the `production` environment. Add no sampling rule, so every request is sent.
5. Choose **Custom Endpoint**: the **endpoint** from step 1, format NDJSON or JSON, and
   the custom header `X-Shruwd-Ingest-Token: YOUR_TOKEN`.
6. Select **Create Drain**. Vercel tests the endpoint, which answers `200`.

> **Warning**
>
> Keep **Team Settings → Security & Privacy → IP Address Visibility** on. With IP
> addresses hidden, Vercel sends no client IP, and no hit can be verified.

## 3. Check the fields

Shruwd reads these fields per line. Common alternative spellings — `ClientIP`,
`clientip`, `remote_addr` — are understood, so most platforms work as they come.
Vercel's format works as it is, including the request details it nests under `proxy`.

| Field | Required | Used for |
|---|---|---|
| `client_ip` | **Yes** | Verifying the hit is really from the crawler it claims to be |
| `user_agent` | **Yes** | Identifying which bot. Lines without one are dropped |
| `timestamp` | No | Defaults to arrival time. ISO 8601 |
| `host` | No | Telling your domains apart |
| `path` | No | Per-page diagnostics |
| `status` | No | Detecting crawlers being refused |
| `method`, `bytes` | No | Filtering and volume |

If your host offers drain sampling, keep it at 100%. Crawler numbers are presented as
exact counts, and a sampled drain would make them partial.

A single line looks like this:

```json
{"timestamp":"2026-09-09T10:00:00Z","host":"example.com","path":"/pricing","user_agent":"Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)","status":200,"client_ip":"23.98.142.176","method":"GET","bytes":18234}
```

## 4. Test before relying on it

Add `?validate=1` to the endpoint to parse a sample and report on it without writing
anything.

```bash
curl -X POST "YOUR_ENDPOINT?validate=1" \
  -H "X-Shruwd-Ingest-Token: YOUR_TOKEN" \
  -H "Content-Type: application/x-ndjson" \
  -d '{"timestamp":"2026-01-01T00:00:00Z","host":"example.com","path":"/","user_agent":"GPTBot","status":200,"client_ip":"1.2.3.4"}'
```

`usable: true` means the endpoint and token are both good. Anything else names what is
missing.

## 5. Confirm it is live

Within a few minutes of real traffic, **Settings → Crawler logs** moves from *awaiting
first logs* to *active*, and **Last received** fills in.

## Response codes

| Code | Meaning |
|---|---|
| `202` | Accepted (`200` to Vercel, which asks for it). Parsing happens afterwards, so this does not mean the lines were usable — use `?validate=1` to check that |
| `401` | Wrong or revoked token |
| `413` | Batch over the size cap |
| `422` | Only with `?validate=1`: the sample can't be used, and `reason` says why |

## Next steps

- [Ingest not arriving](https://shruwd.io/docs/crawler-logs/ingest-not-arriving.md) — if the card stays on *awaiting first logs*
- [Crawlers tracked](https://shruwd.io/docs/crawler-logs/crawlers-tracked.md) — what gets recognised