CiteAura
Start free trial
Engineering Checklist

AI Crawler Access Checklist: GPTBot, Claude-SearchBot, PerplexityBot

Managing crawler access in 2026 requires understanding the exact distinction between model training scrapers, live search indexing bots, and user-initiated page fetchers across OpenAI, Anthropic, Perplexity, and Google. Here is the verified taxonomy and production configuration guide.

Updated 2 September 2026 · 10 min read · Server Configuration

By David Park

The 2026 AI Web Crawler Taxonomy

Major AI providers separate user-agents by functional role, allowing webmasters to permit search indexation while optionally opting out of general pre-training datasets:

User-Agent Token Operator Official Purpose Impact if Blocked
OAI-SearchBot OpenAI Search indexing for ChatGPT Search & SearchGPT Site will not be indexed or cited in ChatGPT Search results
ChatGPT-User OpenAI User-initiated live web browsing requests in ChatGPT ChatGPT cannot browse your site when a user prompts it
GPTBot OpenAI Automated model training data collection Excluded from OpenAI foundation model training sets
Claude-SearchBot Anthropic Search indexing and retrieval for Claude web search Claude search features cannot index or surface your domain
Claude-User Anthropic User-triggered on-demand page fetch (e.g. user submits URL) Claude chat cannot read specific links provided by users
ClaudeBot Anthropic Automated training data collection for Claude models Excluded from Anthropic model training corpora
PerplexityBot Perplexity Web search indexing & answer grounding Excluded from Perplexity search index and citation arrays
Perplexity-User Perplexity User-initiated live search queries Real-time query fallback cannot fetch page
Google-Extended Google Gemini & Vertex AI model training Does NOT affect Google Search or Google AI Overviews

Production-Ready robots.txt Configuration

Below is a balanced configuration that explicitly allows AI search crawlers on public documentation and marketing pages while restricting private application routes. Note that automated search indexers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are governed by robots.txt, whereas user-initiated request bots (ChatGPT-User, Claude-User, Perplexity-User) fetch on explicit user command and generally bypass robots.txt unless blocked at the CDN/WAF layer:

# Allow AI Search Crawlers on Public Routes
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Allow: /docs
Allow: /pricing
Allow: /blog
Allow: /llms.txt
Disallow: /app/
Disallow: /admin/
Disallow: /api/
Disallow: /checkout/

# Optional: Opt out of automated model training datasets
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

Debugging Cloudflare and WAF Blocks

Even with an open robots.txt, your CDN or Web Application Firewall may challenge AI crawlers with HTTP 403 Forbidden or Cloudflare Managed Challenges.

1. Simulating Crawler User-Agents via cURL

Run these terminal commands against your production domain to verify response headers:

# Test OpenAI SearchBot
curl -I -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.0; +https://openai.com/bot)" https://yourdomain.com/

# Test Anthropic SearchBot
curl -I -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claudebot)" https://yourdomain.com/

# Test PerplexityBot
curl -I -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://yourdomain.com/

# Expected response:
# HTTP/2 200
# content-type: text/html; charset=utf-8

2. Cloudflare Security Rules Adjustment

If the response returns HTTP 403 or includes cf-mitigated: challenge:

Automating Crawler Health Monitoring

Crawl rules often regress during frontend redeployments or DNS migrations. CiteAura's AI visibility audit monitors your domain's live crawler status, alerting your engineering team when an allow rule is accidentally broken.

Measurement notice: CiteAura audits technical readiness and sampling provenance; it does not guarantee specific generative model outputs or ranking placements.

Audit your server's robots.txt and WAF clearance for all major AI web crawlers in under 60 seconds.

Start free trial

Sources