Is your website blocking AI?
A single line in your robots.txt file, or a security setting at your hosting company, can stop ChatGPT, Claude or Perplexity reading your site — so they can't recommend you. Enter your address to check every major AI crawler.
Blocked something you shouldn't have?
Our guide has a copy-and-paste robots.txt template that lets the visibility crawlers in and explains the training ones: Which AI bots to let into your website.
About AI crawlers
Which AI crawlers should I allow?
For visibility, allow the search and live-fetch crawlers: OAI-SearchBot and ChatGPT-User (ChatGPT), Claude-SearchBot and Claude-User (Claude), PerplexityBot and Perplexity-User, Googlebot, Bingbot, Applebot and DuckAssistBot. OpenAI says sites that block OAI-SearchBot won't be shown in ChatGPT search answers.
What about GPTBot and Google-Extended?
Those only collect training data. Blocking them is your choice and doesn't remove you from ChatGPT search or Google's AI Overviews — blocking Google-Extended does not take you out of AI Overviews.
My robots.txt allows everything. Could I still be blocked?
Yes. Since July 2025 Cloudflare blocks AI crawlers by default on new sites, and any “checking your browser” page stops crawlers that don't run JavaScript. This tool flags security walls it runs into.
See exactly where you're showing up, and where you're not.
Book a free 30-minute consultation. We'll look at how you're found on Google, Google's AI and ChatGPT before the call, show you what we found, and tell you what we'd fix first. No sales pitch if it's not a fit.
Free · 30 minutes · Google Meet · No obligation