# AI Robots.txt Generator

Most robots.txt generators only know about Googlebot. This one handles every modern AI crawler — OpenAI, Anthropic, Google's Gemini, Perplexity, Apple, and the rest. Toggle each bot, copy the output, ship it.

## Presets

- Allow all (max AI visibility)
- Allow real-time fetches, block training
- Block all AI bots

### OpenAI

**GPTBot**  
Trains future OpenAI models on your content. Block this if you don't want your docs in training data.

- Allow  
- Block

**OAI-SearchBot**  
Indexes for ChatGPT Search results. Block to disappear from ChatGPT's search-style answers.

- Allow  
- Block

**ChatGPT-User**  
Fetched in real-time when a user asks ChatGPT to read your URL. Almost always allow.

- Allow  
- Block

### Anthropic

**ClaudeBot**  
Anthropic's training crawler. Block to keep docs out of future Claude training.

- Allow  
- Block

**anthropic-ai**  
Older Anthropic UA — kept for backward compatibility.

- Allow  
- Block

**Claude-Web**  
Real-time fetches when a Claude user asks it to read a URL.

- Allow  
- Block

### Google

**Google-Extended**  
Controls inclusion in Gemini training and Google's AI overviews. Doesn't affect normal search ranking.

- Allow  
- Block

### Perplexity

**PerplexityBot**  
Indexes for Perplexity answers and sources. Allowing means more citation traffic.

- Allow  
- Block

### Apple

**Applebot-Extended**  
Apple Intelligence training. Doesn't affect Siri search.

- Allow  
- Block

### Other

**CCBot**  
Common Crawl — feeds many AI training datasets indirectly.

- Allow  
- Block

**Bytespider**  
ByteDance / TikTok's crawler. Often misbehaves — many sites block it.

- Allow  
- Block

**FacebookBot**  
Meta's training crawler for Llama. Block to keep out.

- Allow  
- Block

### Sitemap URL (optional)

robots.txt

```plaintext
# Generated by Docsio AI Robots.txt Generator
# https://docsio.co/tools/ai-robots-txt

User-agent: *
Allow: /

# OpenAI
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Anthropic
User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: Claude-Web
Allow: /

# Google
User-agent: Google-Extended
Allow: /

# Perplexity
User-agent: PerplexityBot
Allow: /

# Apple
User-agent: Applebot-Extended
Allow: /

# Other
User-agent: CCBot
Allow: /

User-agent: Bytespider
Disallow: /

User-agent: FacebookBot
Allow: / 
```

## How it works

1. **Toggle**  
   Pick allow or block for each AI bot, or use a preset.
2. **Generate**  
   Output is grouped by provider with a comment header.
3. **Deploy**  
   Save as robots.txt at your site root.