# Cloudflare Worker

A small Worker on a route in front of your site forwards AI-crawler hits to Shruwd. It
runs **before cache lookup**, so it sees the requests Cloudflare answers from cache
without calling your origin — which on a static or prerendered site is most of the pages
worth measuring.

Works on any Cloudflare plan. It sends one short line per crawler hit and nothing at all
for ordinary visitors.

> **Warning**
>
> The Worker only sees requests Cloudflare lets through. A request refused by Cloudflare's
> own settings (AI crawler blocks, WAF custom rules, rate limiting) stops before the Worker
> runs, so Shruwd never hears about it. Cloudflare does not document where Bot Fight Mode
> runs, so do not count on seeing its refusals either. To see what Cloudflare refused, open
> **Security → Analytics** in the Cloudflare dashboard. On an Enterprise zone,
> [Logpush](https://shruwd.io/docs/crawler-logs/cloudflare-logpush.md) records those refusals too.
>
> What the Worker does see is a refusal made behind Cloudflare — by your host, a security
> plugin or your server — and the finding says that is where it happened.

## Prerequisites

- **Permissions:** admin or owner, to create the token.
- A Cloudflare zone for your domain, and permission to deploy a Worker on it.
- `wrangler` available (`npx wrangler` needs no install).

## 1. Get your endpoint and token

1. Open the brand's **Settings → Crawler logs**.
2. Select **Create token**.
3. Copy the **endpoint** and the **token**.

> **Warning**
>
> The token is shown once and cannot be retrieved later. Creating a new one immediately
> revokes the old one, which will stop a Worker that is already running.

## 2. Create the Worker

Make a directory in your site's repository with the two files below.

```jsonc [wrangler.jsonc]
{
  "$schema": "node_modules/wrangler/config-schema.json",
  "name": "shruwd-log-forwarder",
  "main": "src/index.js",
  "compatibility_date": "2026-09-01",

  // The route must match your canonical host ONLY. If customer custom domains
  // resolve through this same zone, this pattern must not catch them.
  "routes": [
    { "pattern": "example.com/*", "zone_name": "example.com" }
  ],

  "vars": {
    "CANONICAL_HOST": "example.com",

    // Comma-separated path prefixes. "/" is the root page only;
    // "/blog/" is everything underneath it.
    "PATH_ALLOWLIST": "/,/pricing,/features,/blog/,/docs/",

    "SHRUWD_INGEST_URL": "YOUR_ENDPOINT"
  }
}
```

```js [src/index.js]
// Forwards AI-crawler hits to Shruwd. Fails open: whatever happens in here,
// your site's response is returned unchanged.

// A broad prefilter, not the bot list. Identification proper happens on
// Shruwd's side against a catalogue that is kept current, so this only has to
// be loose enough not to drop something that catalogue would have caught.
const BOT_UA_PREFILTER =
  /bot|crawl|spider|gpt|claude|perplexity|bytespider|applebot|ccbot|meta-external|google|extended/i

// Static assets are never the page a crawler judges you on. .txt and .xml stay,
// because robots.txt and sitemaps are interesting.
const ASSET_PATH =
  /\.(?:js|mjs|cjs|css|map|png|jpe?g|gif|webp|avif|svg|ico|woff2?|ttf|otf|eot|mp4|webm|mp3|wav|pdf|zip|gz|json|wasm)$/i

function parseAllowlist(value) {
  return String(value ?? '/').split(',').map((s) => s.trim()).filter(Boolean)
}

// Cheapest checks first — this runs on every request to the zone.
function decide({ method, host, path, userAgent }, { canonicalHost, allowlist }) {
  if (method !== 'GET' && method !== 'HEAD') return false
  if (host.toLowerCase() !== canonicalHost.toLowerCase()) return false
  if (ASSET_PATH.test(path)) return false
  if (!allowlist.some((p) => (p === '/' ? path === '/' : path === p || path.startsWith(p)))) return false
  return BOT_UA_PREFILTER.test(userAgent)
}

let warned = false

function maybeReport(request, response, env, ctx) {
  const url = new URL(request.url)
  const userAgent = request.headers.get('user-agent') ?? ''

  const report = decide(
    { method: request.method, host: url.hostname, path: url.pathname, userAgent },
    { canonicalHost: env.CANONICAL_HOST, allowlist: parseAllowlist(env.PATH_ALLOWLIST) },
  )
  if (!report) return

  // Set by the edge; a client cannot forge it. Inside a Worker,
  // x-forwarded-for can be forged, so it is not used as a fallback.
  const clientIp = request.headers.get('cf-connecting-ip')
  if (!clientIp) return

  // The query string is never sent: it carries ids on some routes, and a
  // per-page crawl count needs none of it.
  const line = {
    timestamp: new Date().toISOString(),
    host: url.hostname,
    path: url.pathname,
    user_agent: userAgent,
    status: response.status,
    client_ip: clientIp,
    method: request.method,
    bytes: Number(response.headers.get('content-length') ?? 0) || 0,
  }

  // One request per hit, deliberately. A Worker isolate is ephemeral and
  // per-colo, so a buffer would be lost on eviction and split across colos.
  // Volume is low because the prefilter already ran.
  ctx.waitUntil(
    fetch(env.SHRUWD_INGEST_URL, {
      method: 'POST',
      headers: {
        'x-shruwd-ingest-token': env.SHRUWD_INGEST_TOKEN,
        'content-type': 'application/x-ndjson',
      },
      body: JSON.stringify(line) + '\n',
    })
      .then((res) => {
        if (!res.ok && !warned) {
          warned = true
          console.warn(`shruwd forwarder: ingest answered ${res.status}; check the token and URL`)
        }
      })
      .catch((error) => {
        if (!warned) {
          warned = true
          console.warn('shruwd forwarder: ingest unreachable', error?.message ?? error)
        }
      }),
  )
}

export default {
  async fetch(request, env, ctx) {
    // Never inside the try. The only unconditional path is returning your
    // origin's response — a logging pipeline that can break the site it
    // observes is worse than one that loses events.
    const response = await fetch(request)
    try {
      maybeReport(request, response, env, ctx)
    } catch {
      // deliberately silent
    }
    return response
  },
}
```

## 3. Configure it

Edit the four values in `wrangler.jsonc`:

| Setting | Value |
|---|---|
| `routes` | Your canonical host, e.g. `example.com/*` with `zone_name` `example.com` |
| `CANONICAL_HOST` | Your canonical host, without a scheme |
| `PATH_ALLOWLIST` | Your public, crawlable pages, comma separated |
| `SHRUWD_INGEST_URL` | The **endpoint** from step 1 |

> **Warning**
>
> `PATH_ALLOWLIST` is an allowlist on purpose. Your site probably serves customer content
> and token-bearing links from the same origin, and a blocklist fails open — the next route
> someone adds starts forwarding data to a third party. The worst case here is a missing
> datapoint for a marketing page.

## 4. Deploy

Store the token as a secret rather than putting it in the config or the source.

```bash
npx wrangler secret put SHRUWD_INGEST_TOKEN
npx wrangler deploy
```

Paste the token when prompted.

## 5. Verify

Request a **prerendered** page with a crawler user-agent. That is the case origin
middleware could not see, so it is the one worth testing.

```bash
curl -s -o /dev/null "https://example.com/pricing" \
  -H "User-Agent: Mozilla/5.0 (compatible; GPTBot/1.2)"
```

Within a minute, **Settings → Crawler logs** moves from *awaiting first logs* to
*active*.

Your test hit comes from your own machine, not from OpenAI, so it is recorded as
**unverified** and excluded from headline metrics. That is correct — and the card moving
off *awaiting first logs* is the signal you are looking for.

## Next steps

- [Crawlers tracked](https://shruwd.io/docs/crawler-logs/crawlers-tracked.md) — how verification works
- [Ingest not arriving](https://shruwd.io/docs/crawler-logs/ingest-not-arriving.md) — if nothing shows up