## What are agent-specific robots.txt directives?

Standard robots.txt controls web crawlers. By adding User-agent directives for AI-specific crawlers, you signal that your site is aware of and welcomes AI agents.

## Example

Add to your `/robots.txt`:

```
User-agent: GPTBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: Anthropic-AI
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /
```

## Known AI crawlers

| User-agent | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training and browsing |
| Claude-Web | Anthropic | Web browsing |
| Anthropic-AI | Anthropic | Training |
| PerplexityBot | Perplexity | Search and answers |
| Google-Extended | Google | AI training |
| Applebot | Apple | Siri and AI features |

## Why it matters

Many sites block AI crawlers by default. Explicitly allowing them signals that your content and APIs are designed for agent consumption.

## Spec maturity

**Established convention.** robots.txt is a longstanding web standard (RFC 9309). AI-specific User-agent strings are defined by each AI company.

## Learn more

- [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) — robots.txt specification

## Related

- [llms.txt](/kb/llms-txt)