https://coderlegion.com/tag/bot-management Powered by Question2Answer https://coderlegion.com/32140/cloudflares-ai-bot-controls-and-the-setting-that-can-block-googlebot <p>If Cloudflare sits in front of your site, it may already be choosing which AI bots get through. New domains have been asked about AI crawlers at sign-up since July 2025, and from September 15, 2026 newly added domains start with training and agent bots blocked on pages with ads. This guide explains each feature and the one setting that can quietly block search crawlers.</p> <p>I lay out the timeline, from AI Audit in 2024 to separate Search, Agent and Training policies on every plan in July 2026. Then the trap: since September 2026, any setting that blocks Training also blocks crawlers Cloudflare classifies as both search and training, and its documentation names Googlebot, Applebot and Bingbot as examples. If you depend on Google and Bing, don't block Training on all pages; block training-only crawlers individually instead, and verify with Search Console's live test.</p> <p>I walk through AI Crawl Control's tabs and what the Free plan can and can't detect, Cloudflare's managed robots.txt and Content Signals block, Pay Per Crawl with its HTTP 402 response (still a closed beta), and how verified bots and Web Bot Auth work.</p> <p>There's a step-by-step Free-plan setup for a small business that wants to be found, plus limits: only proxied hostnames are covered, classifications can change, and allowing bots doesn't by itself earn citations.</p> <h4 id="key-takeaways">Key takeaways</h4> <ul> <li>New Cloudflare domains may block AI bots by default</li> <li>Blocking Training can also block Googlebot and Bingbot</li> <li>Block training-only crawlers individually instead</li> <li>Check the robots.txt Cloudflare actually serves</li> <li>Confirm with Search Console and Bing live URL tests</li> </ul> <p>Related on BuiltToWinWeb: <a rel="nofollow" href="https://built2winweb.com/blog/agent-readiness-experiment/">what my own Cloudflare defaults were doing</a></p> <p><strong>Read the full guide on BuiltToWinWeb:</strong> <a rel="nofollow" href="https://built2winweb.com/blog/cloudflare-ai-crawl-control/"><strong>How Cloudflare handles AI crawlers</strong></a></p> <p><em>A summary by Jacob Campbell of an article first published on built2winweb.com on October 7, 2026.</em></p> Articles https://coderlegion.com/32140/cloudflares-ai-bot-controls-and-the-setting-that-can-block-googlebot Fri, 23 Oct 2026 04:01:47 +0000