Author: Harry Sanders
Profile: https://www.linkedin.com/in/harry-sanders/
I've audited 100+ websites this month and this issue comes up constantly
Cloudflare blocks AI crawlers by default on every new website, and I think that's one of the dumbest defaults of the year
Whether it's Google or AI, they're going to get that information from somewhere, unless there's a specific reason you don't want them to have it, you should let them in
If you want AI to find your website and send you traffic, don't disallow the crawlers
Here's how to check in 60 seconds:
1. Open your robots.txt at yoursite.com/robots.txt, look for User-agent entries for GPTBot, PerplexityBot, ClaudeBot, Google-Extended, if any say Disallow: /, you're blocking
2. In Cloudflare, make sure "Block AI training bots" is set to "Do not block" (see screenshot)
3. Quick smoke test: ask Claude or ChatGPT "Can you summarise [yoursite.com]?" if it says it can't access the page, that confirms the block (it may hallucinate, check in periodically)
The fix is usually one toggle, some businesses I've audited were blocking AI crawlers for six months and wondering why they're not appearing in AI searches
And now people spinning up their freshly designed Claude sites via Cloudflare, make sure get your setting sorted
The cheapest SEO win in 2026 is making sure you're actually visible to search engines now and in the future
Cloudflare blocks AI crawlers by default on every new website, and I think that's one of the dumbest defaults of the year
Whether it's Google or AI, they're going to get that information from somewhere, unless there's a specific reason you don't want them to have it, you should let them in
If you want AI to find your website and send you traffic, don't disallow the crawlers
Here's how to check in 60 seconds:
1. Open your robots.txt at yoursite.com/robots.txt, look for User-agent entries for GPTBot, PerplexityBot, ClaudeBot, Google-Extended, if any say Disallow: /, you're blocking
2. In Cloudflare, make sure "Block AI training bots" is set to "Do not block" (see screenshot)
3. Quick smoke test: ask Claude or ChatGPT "Can you summarise [yoursite.com]?" if it says it can't access the page, that confirms the block (it may hallucinate, check in periodically)
The fix is usually one toggle, some businesses I've audited were blocking AI crawlers for six months and wondering why they're not appearing in AI searches
And now people spinning up their freshly designed Claude sites via Cloudflare, make sure get your setting sorted
The cheapest SEO win in 2026 is making sure you're actually visible to search engines now and in the future
Images:
