A lot of businesses are blocking AI from crawling their website right now, and most of them don’t even know it. At TJ Digital, we manage over 40 active client campaigns, and this is one of the most common issues we find during site audits. If AI bots can’t read your pages, AI search engines won’t recommend your business.
There are three main reasons your website might be blocking AI bots. It could be your Cloudflare settings, your hosting provider, or your robots.txt file. I’m going to walk through each one and show you how to check.
Why Are So Many Websites Blocking AI Bots?
Most of them are doing it accidentally. Cloudflare rolled out a one-click AI bot blocking feature in 2024, and over a million websites have turned it on. Hosting providers started throttling AI crawlers to manage server load, and some web developers added blanket bot restrictions to the robots.txt file without understanding the consequences.
The problem is that many of the bots these tools block are the same bots that power AI search. If you block OpenAI’s GPTBot, you’re also blocking ChatGPT from pulling your content into answers.
Your website is the most powerful tool you have to influence what AI tells your potential customers. Blocking those bots is like telling Google not to show your pages in search results.
How Cloudflare Blocks AI Crawlers on Your Website
This is the most common cause. Cloudflare protects roughly 20% of all websites on the internet. As of July 2025, new Cloudflare domains block AI crawlers by default.
The setting was designed to stop AI companies from scraping website content for training purposes. That’s a legitimate concern. But Cloudflare’s blocking doesn’t always distinguish between bots that scrape for training and bots that crawl for search.
If you have Cloudflare’s AI bot protection enabled (or set it up after mid-2025 and never changed the defaults), you could be invisible to ChatGPT, Perplexity, and Google’s AI features.
To fix this, log into your Cloudflare dashboard and go to Security, then Bots. Look for the AI bot controls and make sure search crawlers are allowed. Cloudflare now offers three separate toggles for Search, Training, and Agent crawlers, so you can block training bots while still allowing search bots through.
What to Do If Your Hosting Provider Blocks AI Bots
Your hosting platform might be making this decision for you. I recently consulted for a provider whose platform hosts thousands of websites. They were getting hit by so much AI bot traffic that they decided to block all high-traffic bots, which meant blocking essentially all AI crawlers.
That decision had a detrimental impact on their customers’ ability to show up in AI search results. They’re working on a fix, but not every provider is.
Some hosts handle this well. SiteGround blocks AI training bots by default but allows search bots. Kinsta doesn’t block AI crawlers at all and leaves the decision to you.
Squarespace added a one-click toggle under Settings that’s off by default. But plenty of other providers are blocking everything without telling you.
If your hosting provider is blocking AI bots, talk to them. Ask what their policy is and whether you can opt out.
If they won’t budge, I’d seriously consider moving your website. This is too important to leave to a hosting company’s convenience.
What Should Your Robots.txt File Say About AI Bots?
The third reason your website might be blocking AI is that someone working on your site added restrictions to your robots.txt file. Most websites don’t need to block anything in this file. Truly malicious bots are going to ignore it anyway, so you need to handle those at the server level.
For about 90% of websites, your robots.txt can simply allow all bots. For more control, the most common approach is to block training crawlers while allowing everything else.
Here’s a reference for the most common AI bot user agents:
| Bot Name | Company | Purpose |
| GPTBot | OpenAI | Training data collection |
| OAI-SearchBot | OpenAI | ChatGPT search results |
| ChatGPT-User | OpenAI | ChatGPT browsing for users |
| ClaudeBot | Anthropic | Training data collection |
| Google-Extended | AI training (Gemini) | |
| PerplexityBot | Perplexity | Search and indexing |
If you choose to block training bots, make sure you’re keeping the search bots allowed. Blocking GPTBot stops training, but blocking OAI-SearchBot stops ChatGPT from finding your content when users ask it questions. Those are two different bots with two different jobs.
My recommendation is to let all of them through, including the training bots. We’re close to a world where all the information your customers search for will come from AI. The more those models know about your business, the better they can represent you.
How to Check If AI Can Access Your Website
The easiest test comes straight from the source. Go to ChatGPT, Claude, or Perplexity and give it one of your website URLs. Ask it to summarize what’s on the page.
If it returns accurate information with citations, your site is accessible. If it says it can’t access the page or gives vague, wrong answers, something is blocking it.
For a more technical check, you can simulate an AI bot request from your terminal. Run curl -I -A “GPTBot/1.2” https://yourdomain.com and look for a 200 OK response. A 403 error means that bot is being blocked.
You can swap in different user agent strings like “OAI-SearchBot/1.0” or “ClaudeBot” to test each one individually.
Robots.txt rules and server-level blocking can give different results. Your robots.txt might say “allowed,” but a Cloudflare rule or hosting firewall could still be returning 403 errors. Check both.
What Happens When AI Can’t Crawl Your Site?
When AI search engines can’t access your website, you don’t show up in their answers. No citation, no link, no traffic.
This is the same tradeoff as blocking Googlebot in 2005. If your pages can’t be fetched, you won’t appear in AI-generated answers.
ChatGPT alone has over 100 million weekly users. Google’s AI Mode is replacing traditional search results for millions of queries.
These platforms crawl web content, index it, and use it to generate answers. If your site is blocked, the AI will recommend your competitors who are accessible.
The businesses that show up in AI answers are the ones whose content is accessible, well-structured, and easy for machines to parse. Make sure to prepare your website for AI crawlers now. For more on what AI search looks for, read about how ranking in ChatGPT works.
Contact TJ Digital for a digital marketing audit that includes an AI visibility review. We’ll check whether AI bots can access your site, identify what’s blocking them, and show you exactly what to fix.