Crawler logs

Server middleware

Forward AI-crawler hits from a Nuxt or Nitro origin — only when no CDN answers for your site.

Middleware in your own app forwards AI-crawler hits to Shruwd. Use it only when no CDN answers for your site.

Read this before you start. Middleware only sees requests that reach your origin. If your site is static, prerendered or ISR, a CDN answers most requests without ever calling your server — including exactly the marketing pages worth measuring. On one real Nuxt site, eleven of twenty tracked paths never touched the origin at all.

If a CDN sits in front of you, use the forwarder Worker or a platform drain instead. Both see cached requests; this cannot.

Prerequisites

  • Permissions: admin or owner, to create the token.
  • A Nuxt or Nitro app whose origin genuinely serves the traffic.

1. Get your endpoint and token

Open the brand's Settings → Crawler logs and select Create token. Copy both values — the token is shown once, and creating a new one revokes the old one.

2. Add the middleware

Create server/middleware/crawler-log.ts in your app.

server/middleware/crawler-log.ts
const ENDPOINT = process.env.SHRUWD_INGEST_URL
const TOKEN = process.env.SHRUWD_INGEST_TOKEN

// Flush on whichever comes first. Both are small: a serverless instance can be
// frozen between requests, and anything still buffered is lost.
const MAX_BATCH = 20
const MAX_WAIT_MS = 10_000

// A deliberately broad prefilter. Real identification happens on Shruwd's side
// against a list that is kept current, so this only has to be loose enough not
// to drop something that list would have caught.
const LOOKS_LIKE_BOT = /bot|crawl|spider|gpt|claude|perplexity|bytespider|applebot|ccbot|meta-external/i

// An ALLOWLIST of your public, crawlable pages — not a list of exclusions.
// A blocklist fails open: the next route someone adds starts forwarding
// customer data to a third party. Edit this to match your site.
const LOGGED_PREFIXES = ['/docs', '/features', '/pricing', '/blog', '/faq']

function isLoggedPath(pathname: string): boolean {
  if (pathname === '/') return true
  return LOGGED_PREFIXES.some((p) => pathname === p || pathname.startsWith(p + '/'))
}

let buffer: string[] = []
let timer: ReturnType<typeof setTimeout> | null = null

// Complain once, then stay quiet. Without this, a missing variable, a revoked
// token and "no crawlers yet" all look identical.
let warned = false
function warnOnce(message: string) {
  if (warned) return
  warned = true
  console.warn(`[shruwd] log drain not working: ${message}`)
}

function flush() {
  if (timer) { clearTimeout(timer); timer = null }
  if (buffer.length === 0) return

  if (!ENDPOINT || !TOKEN) {
    warnOnce('SHRUWD_INGEST_URL or SHRUWD_INGEST_TOKEN is not set')
    buffer = []
    return
  }

  const body = buffer.join('\n')
  buffer = []

  // Not awaited: a network stall must never reach your request path.
  fetch(ENDPOINT, {
    method: 'POST',
    headers: { 'X-Shruwd-Ingest-Token': TOKEN, 'content-type': 'application/x-ndjson' },
    body,
  })
    .then((res) => {
      if (!res.ok) warnOnce(`endpoint returned ${res.status}`)
    })
    .catch((error) => {
      warnOnce(`request failed: ${error instanceof Error ? error.message : String(error)}`)
    })
}

export default defineEventHandler((event) => {
  // The query string is dropped, not just ignored — it carries ids on some
  // routes and a per-page crawl count needs none of it.
  const pathname = event.path.split('?')[0] ?? '/'
  if (!isLoggedPath(pathname)) return

  const ua = getRequestHeader(event, 'user-agent') ?? ''
  if (!LOOKS_LIKE_BOT.test(ua)) return

  // cf-connecting-ip first: behind Cloudflare it is the only client IP header
  // a visitor cannot forge. The first x-forwarded-for entry is the fallback.
  const ip =
    getRequestHeader(event, 'cf-connecting-ip') ??
    (getRequestHeader(event, 'x-forwarded-for') ?? '').split(',')[0]?.trim() ??
    event.node.req.socket.remoteAddress ??
    ''

  if (!ip) return

  const started = Date.now()

  // Status and byte count only exist once the response is finished.
  event.node.res.once('finish', () => {
    try {
      buffer.push(JSON.stringify({
        timestamp: new Date(started).toISOString(),
        host: getRequestHeader(event, 'host') ?? '',
        path: pathname,
        user_agent: ua,
        status: event.node.res.statusCode,
        client_ip: ip,
        method: event.method,
        bytes: Number(event.node.res.getHeader('content-length') ?? 0),
      }))

      if (buffer.length >= MAX_BATCH) flush()
      else if (!timer) timer = setTimeout(flush, MAX_WAIT_MS)
    } catch {
      // Never throw into the request path.
    }
  })
})

Edit LOGGED_PREFIXES to your own public pages before deploying. It is an allowlist on purpose: your app probably serves customer content and token-bearing links from the same origin, and none of that should leave your servers.

3. Set the variables

VariableValue
SHRUWD_INGEST_URLthe endpoint from step 1
SHRUWD_INGEST_TOKENthe token from step 1

Set them in your deployed environment, not only locally. That omission is the most common reason nothing arrives.

If you switch to useRuntimeConfig() instead of process.env, two things must change together. Declare the keys in nuxt.config.ts, and prefix the variables with NUXT_ (NUXT_SHRUWD_INGEST_URL). Miss either and the config reads undefined, the middleware returns at its first check, and nothing is ever sent — with no error anywhere.

4. Verify

curl -X POST "YOUR_ENDPOINT?validate=1" \
  -H "X-Shruwd-Ingest-Token: YOUR_TOKEN" \
  -H "Content-Type: application/x-ndjson" \
  -d '{"timestamp":"2026-01-01T00:00:00Z","host":"example.com","path":"/","user_agent":"GPTBot","status":200,"client_ip":"1.2.3.4"}'

usable: true means the endpoint and token are good, so any remaining problem is in your app. Then deploy and wait for real crawler traffic — the buffer flushes at 20 events or 10 seconds, so a single test request will not appear instantly.

Known limits

  • Cached responses are invisible. A request your CDN serves never reaches the middleware. "Zero hits on /" can mean "cached", not "never crawled".
  • Buffered events are lost when a serverless instance is reclaimed. Crawler hits feed rates and per-page counts, so losing a small fraction evenly does not bias them.
  • A brand-new crawler can slip the prefilter until the pattern is widened.

Next steps