# Server middleware

Middleware in your own app forwards AI-crawler hits to Shruwd. Use it only when no CDN
answers for your site.

> **Warning**
>
> **Read this before you start.** Middleware only sees requests that reach your origin. If
> your site is static, prerendered or ISR, a CDN answers most requests without ever calling
> your server — including exactly the marketing pages worth measuring. On one real Nuxt
> site, eleven of twenty tracked paths never touched the origin at all.
>
> If a CDN sits in front of you, use the [forwarder
> Worker](https://shruwd.io/docs/crawler-logs/cloudflare-worker.md) or a
> [platform drain](https://shruwd.io/docs/crawler-logs/platform-log-drains.md) instead. Both see cached
> requests; this cannot.

## Prerequisites

- **Permissions:** admin or owner, to create the token.
- A Nuxt or Nitro app whose origin genuinely serves the traffic.

## 1. Get your endpoint and token

Open the brand's **Settings → Crawler logs** and select **Create token**. Copy both
values — the token is shown once, and creating a new one revokes the old one.

## 2. Add the middleware

Create `server/middleware/crawler-log.ts` in your app.

```ts [server/middleware/crawler-log.ts]
const ENDPOINT = process.env.SHRUWD_INGEST_URL
const TOKEN = process.env.SHRUWD_INGEST_TOKEN

// Flush on whichever comes first. Both are small: a serverless instance can be
// frozen between requests, and anything still buffered is lost.
const MAX_BATCH = 20
const MAX_WAIT_MS = 10_000

// A deliberately broad prefilter. Real identification happens on Shruwd's side
// against a list that is kept current, so this only has to be loose enough not
// to drop something that list would have caught.
const LOOKS_LIKE_BOT = /bot|crawl|spider|gpt|claude|perplexity|bytespider|applebot|ccbot|meta-external/i

// An ALLOWLIST of your public, crawlable pages — not a list of exclusions.
// A blocklist fails open: the next route someone adds starts forwarding
// customer data to a third party. Edit this to match your site.
const LOGGED_PREFIXES = ['/docs', '/features', '/pricing', '/blog', '/faq']

function isLoggedPath(pathname: string): boolean {
  if (pathname === '/') return true
  return LOGGED_PREFIXES.some((p) => pathname === p || pathname.startsWith(p + '/'))
}

let buffer: string[] = []
let timer: ReturnType<typeof setTimeout> | null = null

// Complain once, then stay quiet. Without this, a missing variable, a revoked
// token and "no crawlers yet" all look identical.
let warned = false
function warnOnce(message: string) {
  if (warned) return
  warned = true
  console.warn(`[shruwd] log drain not working: ${message}`)
}

function flush() {
  if (timer) { clearTimeout(timer); timer = null }
  if (buffer.length === 0) return

  if (!ENDPOINT || !TOKEN) {
    warnOnce('SHRUWD_INGEST_URL or SHRUWD_INGEST_TOKEN is not set')
    buffer = []
    return
  }

  const body = buffer.join('\n')
  buffer = []

  // Not awaited: a network stall must never reach your request path.
  fetch(ENDPOINT, {
    method: 'POST',
    headers: { 'X-Shruwd-Ingest-Token': TOKEN, 'content-type': 'application/x-ndjson' },
    body,
  })
    .then((res) => {
      if (!res.ok) warnOnce(`endpoint returned ${res.status}`)
    })
    .catch((error) => {
      warnOnce(`request failed: ${error instanceof Error ? error.message : String(error)}`)
    })
}

export default defineEventHandler((event) => {
  // The query string is dropped, not just ignored — it carries ids on some
  // routes and a per-page crawl count needs none of it.
  const pathname = event.path.split('?')[0] ?? '/'
  if (!isLoggedPath(pathname)) return

  const ua = getRequestHeader(event, 'user-agent') ?? ''
  if (!LOOKS_LIKE_BOT.test(ua)) return

  // cf-connecting-ip first: behind Cloudflare it is the only client IP header
  // a visitor cannot forge. The first x-forwarded-for entry is the fallback.
  const ip =
    getRequestHeader(event, 'cf-connecting-ip') ??
    (getRequestHeader(event, 'x-forwarded-for') ?? '').split(',')[0]?.trim() ??
    event.node.req.socket.remoteAddress ??
    ''

  if (!ip) return

  const started = Date.now()

  // Status and byte count only exist once the response is finished.
  event.node.res.once('finish', () => {
    try {
      buffer.push(JSON.stringify({
        timestamp: new Date(started).toISOString(),
        host: getRequestHeader(event, 'host') ?? '',
        path: pathname,
        user_agent: ua,
        status: event.node.res.statusCode,
        client_ip: ip,
        method: event.method,
        bytes: Number(event.node.res.getHeader('content-length') ?? 0),
      }))

      if (buffer.length >= MAX_BATCH) flush()
      else if (!timer) timer = setTimeout(flush, MAX_WAIT_MS)
    } catch {
      // Never throw into the request path.
    }
  })
})
```

**Edit `LOGGED_PREFIXES`** to your own public pages before deploying. It is an allowlist
on purpose: your app probably serves customer content and token-bearing links from the
same origin, and none of that should leave your servers.

## 3. Set the variables

| Variable | Value |
|---|---|
| `SHRUWD_INGEST_URL` | the endpoint from step 1 |
| `SHRUWD_INGEST_TOKEN` | the token from step 1 |

Set them in your **deployed** environment, not only locally. That omission is the most
common reason nothing arrives.

> **Warning**
>
> If you switch to `useRuntimeConfig()` instead of `process.env`, two things must change
> together. Declare the keys in `nuxt.config.ts`, **and** prefix the variables with `NUXT_`
> (`NUXT_SHRUWD_INGEST_URL`). Miss either and the config reads `undefined`, the middleware
> returns at its first check, and nothing is ever sent — with no error anywhere.

## 4. Verify

```bash
curl -X POST "YOUR_ENDPOINT?validate=1" \
  -H "X-Shruwd-Ingest-Token: YOUR_TOKEN" \
  -H "Content-Type: application/x-ndjson" \
  -d '{"timestamp":"2026-01-01T00:00:00Z","host":"example.com","path":"/","user_agent":"GPTBot","status":200,"client_ip":"1.2.3.4"}'
```

`usable: true` means the endpoint and token are good, so any remaining problem is in your
app. Then deploy and wait for real crawler traffic — the buffer flushes at 20 events or
10 seconds, so a single test request will not appear instantly.

## Known limits

- **Cached responses are invisible.** A request your CDN serves never reaches the
  middleware. "Zero hits on `/`" can mean "cached", not "never crawled".
- **Buffered events are lost** when a serverless instance is reclaimed. Crawler hits feed
  rates and per-page counts, so losing a small fraction evenly does not bias them.
- **A brand-new crawler can slip the prefilter** until the pattern is widened.

## Next steps

- [Ingest not arriving](https://shruwd.io/docs/crawler-logs/ingest-not-arriving.md) — step-by-step diagnosis
- [Crawlers tracked](https://shruwd.io/docs/crawler-logs/crawlers-tracked.md) — what gets recognised