Crawler logs

Cloudflare Logpush

Point a Logpush job at Shruwd to send your zone request logs with no code to deploy.

A Logpush job with an HTTP destination sends your zone's request logs straight to Shruwd. No code, and it captures everything the edge serves.

Prerequisites

Permissions: admin or owner.
A Cloudflare Enterprise zone. Logpush is not available on Free, Pro or Business — on those plans use the forwarder Worker instead.

1. Get the destination URL

Open the brand's Settings → Crawler logs.
Select Create token.
Copy the Logpush destination URL. It already has the token embedded as a header_Authorization parameter, which Logpush turns into a request header on every upload.

The URL contains the token and is shown once. Creating a new token revokes the old one and will stop a job that is already running.

2. Create the job

In the Cloudflare dashboard, go to your zone → Analytics & Logs → Logpush → Create a Logpush job.

Step	Value
Destination	HTTP destination
HTTP endpoint	the destination URL from step 1, verbatim
Dataset	HTTP requests
If logs match	`ClientRequestHost` equals your domain, or `ClientRequestHost` equals it with `www.` in front
Fields	`EdgeStartTimestamp`, `ClientRequestHost`, `ClientRequestPath`, `ClientRequestUserAgent`, `EdgeResponseStatus`, `ClientIP`, `ClientRequestMethod`, `EdgeResponseBytes`
Advanced → timestamp format	Any. RFC3339, Unix and UnixNano are all read
Advanced → sampling	100%

On Submit, Cloudflare uploads a small gzipped test file and expects a 2xx. Shruwd answers it and does not count it toward your usage.

If job creation fails with error validating destination, the token in the URL is wrong or has been revoked.

3. On a busy zone, cap the batch size

Shruwd reads up to 48 MB decoded per upload and drops the remainder of anything larger. On a high-traffic zone, create the job through the API with explicit limits instead.

curl -X POST "https://api.cloudflare.com/client/v4/zones/YOUR_ZONE_ID/logpush/jobs" \
  -H "Authorization: Bearer YOUR_CLOUDFLARE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "shruwd-crawler-logs",
    "dataset": "http_requests",
    "destination_conf": "YOUR_DESTINATION_URL",
    "enabled": true,
    "filter": "{\"where\":{\"or\":[{\"key\":\"ClientRequestHost\",\"operator\":\"eq\",\"value\":\"example.com\"},{\"key\":\"ClientRequestHost\",\"operator\":\"eq\",\"value\":\"www.example.com\"}]}}",
    "output_options": {
      "field_names": ["EdgeStartTimestamp","ClientRequestHost","ClientRequestPath","ClientRequestUserAgent","EdgeResponseStatus","ClientIP","ClientRequestMethod","EdgeResponseBytes"],
      "timestamp_format": "rfc3339"
    },
    "max_upload_bytes": 5000000,
    "max_upload_records": 10000
  }'

$body = @{
  name             = "shruwd-crawler-logs"
  dataset          = "http_requests"
  destination_conf = "YOUR_DESTINATION_URL"
  enabled          = $true
  filter           = '{"where":{"or":[{"key":"ClientRequestHost","operator":"eq","value":"example.com"},{"key":"ClientRequestHost","operator":"eq","value":"www.example.com"}]}}'
  output_options   = @{
    field_names      = @("EdgeStartTimestamp","ClientRequestHost","ClientRequestPath","ClientRequestUserAgent","EdgeResponseStatus","ClientIP","ClientRequestMethod","EdgeResponseBytes")
    timestamp_format = "rfc3339"
  }
  max_upload_bytes   = 5000000
  max_upload_records = 10000
} | ConvertTo-Json -Depth 5

Invoke-RestMethod -Method Post `
  -Uri "https://api.cloudflare.com/client/v4/zones/YOUR_ZONE_ID/logpush/jobs" `
  -Headers @{ Authorization = "Bearer YOUR_CLOUDFLARE_API_TOKEN" } `
  -ContentType "application/json" -Body $body

4. Verify

Within a few minutes of traffic, Settings → Crawler logs moves from awaiting first logs to active.

A hit from an IP outside the crawler vendor's published ranges is recorded as unverified and excluded from headline metrics. That is correct behaviour, and it still proves the pipe is working.

Next steps

Crawlers tracked — how verification works
Ingest not arriving — if nothing shows up