On 15 September 2026, Cloudflare retired its Block AI Bots toggle and replaced it with three settings, one each for AI search, AI training and AI agents. Cloudflare's advice to existing customers is "Nothing, in almost every case", because old choices carried over. Two changes are still worth checking. Choosing Block for training now shuts out Googlebot, Bingbot and Applebot as well. And sites that had the old toggle on now refuse AI agents such as ChatGPT-User on pages that show ads. Your server's logs won't show these refusals, because they happen at Cloudflare.
I checked the five highest-ranked guides across three searches about these settings on 21 September 2026 (US, desktop), leaving out Cloudflare's own pages, news reports, forums and videos. None tells you where to see the requests Cloudflare turned away. One sends you to your server logs, which can't show a refusal made at Cloudflare.
Find out what the answers say about you.
One measurement, your own crawler data, and the three access findings that matter most. Free, no card, about two minutes.
Start freeWhat Cloudflare changed on 15 September
The old toggle blocked "verified bots that are classified as crawling for the purpose of AI training", and it left alone crawlers that serve search and training at once, such as Googlebot. Cloudflare's announcement replaces it with three settings, each covering one kind of AI traffic:
| Setting | What it covers | Examples, as Cloudflare classes them |
|---|---|---|
| Search | Crawlers that index your pages to answer questions later | OAI-SearchBot, PerplexityBot, Claude-SearchBot |
| Training | Crawlers that collect content to train models | GPTBot, ClaudeBot |
| Agent | Fetches made for a person, right now | ChatGPT-User, browser-use agents |
There are four values: Allow, Disallow AI Training (Training only), Block on pages with ads, and Block. Googlebot, Bingbot and Applebot crawl for search and training at the same time, and Cloudflare treats them as a group of their own.
Cloudflare's help page for the old toggle was last updated on 1 July and still describes the plan announced then. The September announcement is the one that matches how the settings work now.
Setting Training to Block now shuts out Googlebot
From the announcement: "Block and Block on pages with ads now apply to mixed-use crawlers, including Applebot, Bingbot, and Googlebot, so either setting impacts search as well as training."
The option that keeps search and opts you out of training is Disallow AI Training. It publishes a no-training preference in your robots.txt, keeps those three crawlers allowed for search, and blocks every other training crawler, including OpenAI's GPTBot.
Advice written before 15 September to "block AI bots" was about a toggle that left Googlebot alone. Choosing Block under the new settings doesn't, and it can take you out of Google, Bing and Apple search along with the AI crawlers.
If you had Block AI Bots turned on
Cloudflare moved sites that never used the newer settings according to their old choice:
| Old toggle | Search | Training | Agent |
|---|---|---|---|
| Disabled | Allow | Allow | Allow |
| Block | Allow | Disallow AI Training | Block on pages with ads |
| Block on pages with ads | Allow | Disallow AI Training | Block on pages with ads |
Search and training behave much as they did. The new part is the Agent column. Cloudflare's July post counts "chat fetch bots (e.g., ChatGPT-User)" as agents, and the September announcement says Block on pages with ads refuses them "only on pages detected to be serving an ad".
ChatGPT-User is the fetch ChatGPT makes when a person's question sends it to your page. How to rank in ChatGPT covers it alongside OAI-SearchBot and GPTBot. If your pages carry ads, ChatGPT can no longer read them for that person. If they don't, the Agent change should do nothing on your site.
Cloudflare verified bots: verified no longer means allowed by default
Cloudflare verifies bots whose operators show that they identify themselves honestly and don't abuse the access. Being verified used to be enough to get in. From the July post: "Previously, all Verified bots were allowed by default, which was reflected in our basic Bot Fight Mode offering". Cloudflare now says it is "no longer viewing Verified as 'default allowed.'" The setting for the bot's category decides instead. A verified OAI-SearchBot gets in when Search is on Allow, unless another setting or rule of yours stops it.
Bot Fight Mode and AI crawlers
Bot Fight Mode is Cloudflare's free bot protection. It "Issues computationally expensive challenges" to traffic it takes for bots. Two details matter for AI crawlers. It used to let verified bots through by default, and the category settings now decide that. And it "Cannot be customized, adjusted, or reconfigured via WAF custom rules", so you can't write a rule elsewhere that makes an exception to it.
Super Bot Fight Mode, on Pro plans and above, "adds configurable actions per bot category". If Bot Fight Mode is challenging a crawler you want, your choices are to turn it off or to move to a plan where you can set the action per category.
Why your server logs won't show it
When a Cloudflare rule blocks or challenges a crawler, the request stops at Cloudflare. It never reaches your server, so your server's access log has no line for it.
When a rule did the refusing, a Cloudflare Worker doesn't see it either. AI Crawl Control "uses WAF custom rules to block" the crawlers you choose, those rules run before a request reaches a Worker, and when a rule blocks or challenges a request, Cloudflare's docs say "the request stops and does not reach later phases". A Worker that forwards your logs reports only the crawlers that got past your rules.
Cloudflare's own analytics do record the refusals, so that's where the check starts.
How to check what Cloudflare turns away
1. Read your settings
In the Cloudflare dashboard, open your domain's Security Settings. Read the Search, Training and Agent settings, then filter by Bot traffic to see whether Bot fight mode is on. For a site that wants to be cited, aim for Search and Agent on Allow, and Training on Allow or Disallow AI Training. Any Block is worth a second look.
Disallow AI Training works by publishing a preference in your robots.txt, so it shows up in that file. A block on pages with ads can't be written into robots.txt, so only Cloudflare's analytics show that one. Our free scan checks your robots.txt for rules that name AI crawlers.
2. Open AI Crawl Control
AI Crawl Control is available on every plan. It counts requests per crawler, "including blocked and error responses". On the free plan it "only displays metrics for the past 24 hours", so look at a crawler you'd expect every day, such as OAI-SearchBot. Cloudflare adds that blocks from other rules "may not be visible in AI Crawl Control analytics".
3. Check Security Analytics
Security Analytics shows which requests Cloudflare's security products blocked or challenged. The free plan shows up to the last seven days. Look for OAI-SearchBot, ChatGPT-User and PerplexityBot among them.
4. Check your own rules
Cloudflare applies your custom rules and rate limits too. A rule that matches "bot" anywhere in the user agent matches OAI-SearchBot as well, and a rate limit set for human visitors can catch a crawler that fetches many pages quickly.
5. Then check your server
A request Cloudflare let through can still be refused by your server, a security plugin or your host. This PowerShell counts each AI crawler's requests in your access log by status code:
$log = ".\access.log"
$bots = 'OAI-SearchBot|ChatGPT-User|GPTBot|PerplexityBot|Perplexity-User|Claude-SearchBot|Claude-User|ClaudeBot'
Select-String -Path $log -Pattern $bots |
ForEach-Object {
$status = if ($_.Line -match '" (\d{3}) ') { $Matches[1] } else { '?' }
[pscustomobject]@{ Crawler = $_.Matches[0].Value; Status = $status }
} |
Group-Object Crawler, Status |
Sort-Object Name |
Format-Table Count, Name -AutoSize
A row reading 2 ChatGPT-User, 403 means your server turned that crawler away twice. The script expects the common log layout, where the status code follows the quoted request. A log laid out differently shows ? where the code should be.
6. Check it's the real crawler before you change anything
A user agent is only a claim. A request calling itself GPTBot from outside OpenAI's published ranges is probably a scraper, and refusing it is your firewall doing its job. Our AI crawler list explains how to check an address against each vendor's published ranges.
If you're not on Cloudflare
Other hosts have their own switches. Vercel's firewall has an AI Bots Managed Ruleset that is "inactive by default", and its Deny action "blocks all traffic identified as coming from AI bots". Vercel's page doesn't name the bots it covers, so assume Deny may include the search crawlers you want. Security plugins and host firewalls can block crawlers too. Wherever the block sits, look for that product's log of what it refused.
What Shruwd sees
Shruwd, the tool we are building, reads your crawler logs and raises a finding when a verified AI crawler is refused a page on at least three days while a normal browser can load it. What it sees depends on where the logs come from, and choosing a log path covers the options. Cloudflare Logpush, on Enterprise plans, records every request Cloudflare's edge serves. A Worker sees only the requests that got past Cloudflare's rules, so if you send logs that way, check Security Analytics as well.
Frequently asked questions
Does Cloudflare block AI bots by default?
It depends on when the domain joined and what was chosen. In July 2025 Cloudflare announced it would block AI crawlers by default, asking every new domain at sign-up whether to allow them. From 15 September 2026, new domains are offered presets instead: a site without ads is offered Allow for search, training and agents, and an ad-supported site is offered Disallow AI Training plus a block on agents on pages with ads. Existing settings carried over.
Does blocking AI bots on Cloudflare affect Google search?
It can now. Since 15 September 2026, the Block and Block on pages with ads settings apply to Googlebot, Bingbot and Applebot, so they affect search as well as training. Disallow AI Training is the option that keeps search and opts you out of training. The old Block AI Bots toggle left those crawlers alone.
Does Cloudflare block ChatGPT?
Only where a setting says so. Cloudflare classes OpenAI's GPTBot as training, OAI-SearchBot as search, and ChatGPT-User, which fetches a page when a person's question calls for it, as an agent. Sites that had Block AI Bots on, and never used the newer settings, were moved to a setting that refuses agents such as ChatGPT-User on pages with ads, and a new site that says it runs ads is offered the same. Search stays allowed in both cases.
How do I see which AI crawlers Cloudflare blocked?
In the Cloudflare dashboard. AI Crawl Control counts requests per crawler, blocked ones included, and keeps 24 hours on the free plan. Security Analytics shows what Cloudflare's security products blocked or challenged, for up to seven days on the free plan. Your server's own logs won't show these refusals.
