AI companies send two kinds of bot to read websites. Some fetch your pages so an assistant can recommend you today; others collect content to train future models. Block the first kind and you can quietly drop out of AI answers. Here is which is which, from the companies’ own documentation, plus a template you can copy.
What is robots.txt?
A robots.txt file is a short text file at the root of your website, such as www.example.co.uk/robots.txt, that tells bots which parts of the site they may visit. In Google’s words, it “tells search engine crawlers which URLs the crawler can access on your site”. Two things to know. It is allow-by-default: if no rule blocks a bot, it can crawl. And it isn’t security: Google says it “is not a mechanism for keeping a web page out of Google”, so anything private belongs behind a password.
What is the difference between AI search bots and AI training bots?
Search and assistant bots fetch pages to answer questions now, and they decide whether your business can be found, quoted and linked. Training bots collect content that may be used to build future AI models. The two are controlled separately; as OpenAI puts it, “Each setting is independent of the others.” Blocking a training bot doesn’t remove you from AI search, and allowing one doesn’t get you into it.
Which OpenAI bots should I allow?
- OAI-SearchBot: builds the index behind ChatGPT search. OpenAI says “sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers”. Allow it.
- ChatGPT-User: visits a page when a ChatGPT user asks about it. Because a person triggered the visit, OpenAI says “robots.txt rules may not apply”. Allow it anyway.
- GPTBot: collects content that may be used to train OpenAI’s models. It doesn’t affect search. Your choice.
OpenAI says its search systems take about 24 hours to adjust after you change robots.txt.
Which Claude bots should I allow?
- Claude-SearchBot: improves Claude’s search results. Anthropic says blocking it “may reduce your site’s visibility and accuracy in user search results”. Allow it.
- Claude-User: fetches pages when a Claude user asks a question. Blocking it “may reduce your site’s visibility for user-directed web search”. Allow it.
- ClaudeBot: collects content that could contribute to training Anthropic’s models. Your choice.
Anthropic says its bots honour robots.txt, including the non-standard Crawl-delay rule if you want them to visit less often.
Which Perplexity bots should I allow?
- PerplexityBot: “designed to surface and link websites in search results on Perplexity”, and not used to train AI foundation models. Allow it.
- Perplexity-User: fetches pages when a user asks. Perplexity says it “generally ignores robots.txt rules” because a person requested the visit. Allow it.
Perplexity says changes can take up to 24 hours to show.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended isn’t a separate crawler but a robots.txt label that controls whether content Google has crawled can be used to train future Gemini models and to ground Gemini’s answers. Google states it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search”. AI Overviews and AI Mode are part of Google Search, so they are governed by Googlebot, Google’s normal crawler, by your snippet settings and, since 31 August 2026, by the ‘Search generative AI’ setting in Google Search Console. It includes your site by default; switching to exclude removes your pages from AI Overviews, AI Mode and AI features in Discover, but not from normal results.
There is a catch. Google says the same label covers grounding in the Gemini app, meaning content fed to the model at the moment someone asks a question, so blocking it can keep your pages out of what Gemini draws on. If you want Gemini to recommend you, leave it allowed. And never block Googlebot unless you want to vanish from Google altogether.
Which Bing and Apple bots should I allow?
- Bingbot: Bing’s crawler. Bing powers Microsoft Copilot’s web answers, and Microsoft is one of the search providers OpenAI names for ChatGPT. Allow it.
- Applebot: powers search features in Siri, Spotlight and Safari. If your robots.txt mentions Googlebot but not Applebot, Apple says Applebot “will follow Googlebot instructions”. Allow it.
- Applebot-Extended: doesn’t crawl at all. It tells Apple whether content Applebot collected may be used to train Apple’s foundation models, and pages that disallow it “can still be included in search results”. Your choice.
More on how each assistant searches in which AI uses which search engine.
What about DuckDuckGo, Meta and Common Crawl?
- DuckAssistBot: crawls pages in real time for DuckDuckGo’s AI-assisted answers. DuckDuckGo says “this data is not used in any way to train AI models”, and that a block takes 72 hours to apply. Allow it.
- meta-externalagent: Meta says it “crawls the web for use cases such as training foundation AI models”. Your choice.
- Meta-WebIndexer: Meta says it “navigates the web to improve Meta AI search result quality for users”. Allow it.
- OAI-AdsBot: OpenAI’s newest bot only checks the landing pages of ads shown in ChatGPT and isn’t used for training. It only matters if you advertise there.
- CCBot: Common Crawl’s crawler. Common Crawl is a non-profit that publishes a free, open archive of the web, a common source of AI training data. Your choice.
Should a small business block AI training bots?
It’s a business decision, not a technical one. For most local businesses we’d allow them: your website is public marketing material, and being part of what models learn about your area is a small upside. If you publish something you’d rather keep out of AI models, such as paid course material, blocking training bots by name in robots.txt is reasonable and won’t stop search bots finding you. Avoid blanket blocking switches, which can catch search crawlers too.
Can snippet settings stop AI quoting me?
Yes. Google says that to appear as a supporting link in AI Overviews or AI Mode, “a page must be indexed and eligible to be shown in Google Search with a snippet”. Its snippet controls apply to AI features too: nosnippet stops Google showing any text from a page, data-nosnippet hides one section, and max-snippet caps how much text it can quote. These are small tags in your page code, sometimes added by SEO plugins or developers. If they are set, AI features have less, or nothing, to quote, so leave snippets unrestricted on pages you want found.
Could something other than robots.txt be blocking AI bots?
Yes, and it is often the bigger risk, because robots.txt allows everything unless told otherwise. Security services and firewalls can turn bots away before they ever see the file. On 1 July 2025, Cloudflare began blocking AI crawlers by default on new domains. In July 2026 it introduced separate Search, Agent and Training settings. Since 15 September 2026, its Training block, including the older “Block AI bots” option, also applies to crawlers that do search and training together, which Cloudflare said include Googlebot, Applebot and Bingbot. Its defaults for new sites have changed more than once, so check the setting rather than assume.
Security plugins, hosting firewalls and ‘checking your browser’ pages can have the same effect, because a bot that can’t run JavaScript can’t pass a JavaScript challenge. We cover that in can ChatGPT read your website?
How do I check which AI bots can reach my site?
Check five places: your robots.txt file, your page code, Google Search Console, your security settings and your server logs.
- Read your robots.txtType your domain followed by /robots.txt into a browser. Look for “Disallow: /” under any AI bot’s name, or under “User-agent: *”.
- Check your snippet settingsView a page’s source and search for “nosnippet” and “max-snippet”.
- Check Search ConsoleUnder Settings > Search generative AI, make sure your site is set to include. Excluding it takes you out of AI Overviews and AI Mode.
- Check your security serviceIf you use Cloudflare or similar, review its bot and AI crawler settings, confirm search crawlers are allowed and make sure no training block is catching Googlebot, Bingbot or Applebot.
- Look at your server logsIf you or your developer can see them, search for OAI-SearchBot, Claude-SearchBot and PerplexityBot and confirm they get normal ‘200’ responses rather than errors.
Or run our free Is my website blocking AI? check.
What should my robots.txt say?
Use a visibility-first file. This template allows the search and assistant bots that decide whether AI can find you, allows the training bots for brand recall (change those to “Disallow: /” if you decide otherwise) and keeps two private folders out of the general crawl. Googlebot is covered by the final “User-agent: *” section. Replace the sitemap address with your own.
# --- AI search & assistant bots: ALLOW (these gate visibility) ---
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Applebot
User-agent: DuckAssistBot
User-agent: Bingbot
User-agent: Meta-WebIndexer
# Google-Extended also controls Gemini app answers: keep it allowed
User-agent: Google-Extended
Allow: /
Disallow: /admin/
Disallow: /api/
# --- Training bots: ALLOW for brand recall (your choice) ---
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Allow: /
Disallow: /admin/
Disallow: /api/
# --- Everything else ---
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Sitemap: https://www.example.co.uk/sitemap.xmlTwo newer controls to leave alone if you want to be found. Google and Microsoft plan to support a shared AI-training line in robots.txt from 2027 (“User-agent: __usage_ai-training”); Google says a site that blocks it without a separate Google-Extended rule will be treated as blocking Google-Extended, which keeps you out of Gemini app answers. And Microsoft’s page-level “ai-grounding:disallow” tag reduces how often Copilot can use your pages. Also bear in mind that fetchers acting on a person’s request, such as Google-Agent, ChatGPT-User and Perplexity-User, may not follow robots.txt at all.
How you edit the file depends on how your site is built, so ask your developer or platform if you can’t find it.
Frequently asked questions
Do AI bots have to obey robots.txt?
No, it’s a voluntary standard. OpenAI, Anthropic, Perplexity, Google, Apple and DuckDuckGo all document how their crawlers respond to it, but fetchers acting on a user’s request, such as ChatGPT-User, Perplexity-User and Meta-ExternalFetcher, may not follow it. Bots that ignore it have to be stopped by a firewall.
What happens if I don’t have a robots.txt file?
Bots treat the whole site as open. Google says it treats a missing file “as if a valid robots.txt file didn’t exist”, meaning no crawl restrictions. That is fine for AI visibility; a file simply lets you point bots to your sitemap and keep private folders out.
Does an llms.txt file do the same job?
No. llms.txt is a proposed file meant to guide AI models, but no major AI system has been shown to read it: Ahrefs found that 97% of the roughly 38,000 sites with one received no requests for it in May 2026. Robots.txt is the file bots actually check. See our guide to AEO myths.
How quickly do changes take effect?
OpenAI says about 24 hours for ChatGPT search, Perplexity says up to 24 hours, and DuckDuckGo says 72 hours for DuckAssistBot to stop crawling after a block.
Sources
- Google Search Central — Introduction to robots.txt
- Google — How Google interprets the robots.txt specification
- OpenAI — Overview of OpenAI crawlers
- OpenAI — Searching the web with ChatGPT (help centre)
- Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler? (Apr 2026)
- Perplexity — Perplexity crawlers
- Google — Google’s common crawlers (Google-Extended)
- Google Search Central — AI features and your website
- Microsoft Learn — Data, privacy and security for web search in Microsoft Copilot
- Apple — About Applebot (Sep 2026)
- DuckDuckGo — DuckAssistBot
- Meta — Meta web crawlers
- Common Crawl — CCBot
- Cloudflare — Cloudflare just changed how AI crawlers scrape the internet-at-large (press release, Jul 2025)
- Cloudflare blog — Your site, your rules: new AI traffic options for all customers (Jul 2026)
- Ahrefs — We analysed 137K sites: 97% of llms.txt files never get read (Jun 2026)