AI Robots.txt Generator

Most robots.txt generators only know about Googlebot. This one handles every modern AI crawler — OpenAI, Anthropic, Google's Gemini, Perplexity, Apple, and the rest. Toggle each bot, copy the output, ship it.

Presets

  • Allow all (max AI visibility)
  • Allow real-time fetches, block training
  • Block all AI bots

OpenAI

GPTBot
Trains future OpenAI models on your content. Block this if you don't want your docs in training data.

  • Allow
  • Block

OAI-SearchBot
Indexes for ChatGPT Search results. Block to disappear from ChatGPT's search-style answers.

  • Allow
  • Block

ChatGPT-User
Fetched in real-time when a user asks ChatGPT to read your URL. Almost always allow.

  • Allow
  • Block

Anthropic

ClaudeBot
Anthropic's training crawler. Block to keep docs out of future Claude training.

  • Allow
  • Block

anthropic-ai
Older Anthropic UA — kept for backward compatibility.

  • Allow
  • Block

Claude-Web
Real-time fetches when a Claude user asks it to read a URL.

  • Allow
  • Block

Google

Google-Extended
Controls inclusion in Gemini training and Google's AI overviews. Doesn't affect normal search ranking.

  • Allow
  • Block

Perplexity

PerplexityBot
Indexes for Perplexity answers and sources. Allowing means more citation traffic.

  • Allow
  • Block

Apple

Applebot-Extended
Apple Intelligence training. Doesn't affect Siri search.

  • Allow
  • Block

Other

CCBot
Common Crawl — feeds many AI training datasets indirectly.

  • Allow
  • Block

Bytespider
ByteDance / TikTok's crawler. Often misbehaves — many sites block it.

  • Allow
  • Block

FacebookBot
Meta's training crawler for Llama. Block to keep out.

  • Allow
  • Block

Sitemap URL (optional)

robots.txt

# Generated by Docsio AI Robots.txt Generator
# https://docsio.co/tools/ai-robots-txt

User-agent: *
Allow: /

# OpenAI
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Anthropic
User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: Claude-Web
Allow: /

# Google
User-agent: Google-Extended
Allow: /

# Perplexity
User-agent: PerplexityBot
Allow: /

# Apple
User-agent: Applebot-Extended
Allow: /

# Other
User-agent: CCBot
Allow: /

User-agent: Bytespider
Disallow: /

User-agent: FacebookBot
Allow: /

How it works

  1. Toggle
    Pick allow or block for each AI bot, or use a preset.
  2. Generate
    Output is grouped by provider with a comment header.
  3. Deploy
    Save as robots.txt at your site root.