robots.txt for Agents
What are agent-specific robots.txt directives?
Standard robots.txt controls web crawlers. By adding User-agent directives for AI-specific crawlers, you signal that your site is aware of and welcomes AI agents.
Example
Add to your /robots.txt:
User-agent: GPTBot
Allow: /
User-agent: Claude-Web
Allow: /
User-agent: Anthropic-AI
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Known AI crawlers
| User-agent | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training and browsing |
| Claude-Web | Anthropic | Web browsing |
| Anthropic-AI | Anthropic | Training |
| PerplexityBot | Perplexity | Search and answers |
| Google-Extended | AI training | |
| Applebot | Apple | Siri and AI features |
Why it matters
Many sites block AI crawlers by default. Explicitly allowing them signals that your content and APIs are designed for agent consumption.
Spec maturity
Established convention. robots.txt is a longstanding web standard (RFC 9309). AI-specific User-agent strings are defined by each AI company.
Learn more
- RFC 9309 — robots.txt specification