AI crawlers and robots.txt: what to allow and what to block
AI companies run separate crawlers for training their models and for search (finding pages to cite in answers). Blocking a training crawler keeps your content out of future models; blocking a search crawler removes you from that assistant's answers. Most brands that want AI visibility should allow the search crawlers and decide on training separately.
Updated · Citable
The main AI crawlers
| User agent | Company | Purpose | Block it if… |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search | You do not want to appear in ChatGPT search answers |
| GPTBot | OpenAI | Collects content to train models | You do not want your content used for training |
| ChatGPT-User | OpenAI | Fetches a page when a user asks | Rarely; robots.txt may not apply, per OpenAI |
| PerplexityBot | Perplexity | Builds Perplexity's search index (not training, per Perplexity) | You do not want to appear in Perplexity |
| Perplexity-User | Perplexity | Fetches a page for a user's question | Generally ignores robots.txt, per Perplexity |
| Claude-SearchBot | Anthropic | Indexes pages for Claude's search | You do not want to appear in Claude search results |
| Claude-User | Anthropic | Fetches pages when a Claude user asks | You do not want Claude to read your pages for users |
| ClaudeBot | Anthropic | Collects content for model training | You do not want your content used for training |
| Google-Extended | A token controlling Gemini training and some grounding, not a separate crawler | You do not want Gemini trained on your content | |
| Googlebot / Bingbot | Google / Microsoft | Search indexing, which also feeds AI Overviews, Gemini and Copilot | Never, if you want search traffic |
| CCBot | Common Crawl | Open web archive widely used for AI training | You do not want to be in open training datasets |
| Applebot-Extended | Apple | Controls use of Applebot data for Apple's AI training | You do not want Apple's models trained on your content |
A robots.txt for brands that want AI visibility
If your goal is to be found and cited, allow everything. The default User-agent: * / Allow: / already does. If you want to stay in AI search but opt out of training, use:
# Search: stay visible in AI answers
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
# Training: opt out
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlCheck your CDN and firewall
Cloudflare, Vercel, Akamai and others offer bot management with AI-specific switches. Some block AI crawlers by default on new sites or plans. If you want AI visibility, review these settings and allow at least the search crawlers. Your CDN's logs will show whether OAI-SearchBot, PerplexityBot and Claude-SearchBot are being served pages or getting 403s.
Verify it works
- Fetch
yoursite.com/robots.txtand read it as a crawler would: rules are matched per user agent, most specific group wins. - Search server or CDN logs for the user agents above and their status codes.
- OpenAI says robots.txt changes for search take about 24 hours to take effect.
- User agents can be spoofed. Vendors publish IP ranges (OpenAI and Perplexity do) if you need to verify a request is genuine.
Questions people ask
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is for training. ChatGPT search uses OAI-SearchBot. You can block one and allow the other.
Does Google-Extended affect AI Overviews?
Google does not list it as a control for AI Overviews or AI Mode; those follow normal Googlebot and snippet rules. Google-Extended covers Gemini training and grounding in some other products.
Should I block AI crawlers to protect my content?
Publishers who sell content often block training crawlers. Brands that want to be recommended usually should not block search crawlers, because that removes them from AI answers.