AI agent user-agents: the reference list

The major AI crawlers and fetchers, one page each: the exact user-agent string, whether it respects robots.txt, how to allow or block it, and how to verify the traffic is genuine. The key distinction is the three-tier system every vendor runs — training crawlers (collect content for model training, respect robots.txt), search-index crawlers (build the index behind cited answers, respect robots.txt), and user-initiated fetchers (fetch live because a human asked; most vendors state these ignore robots.txt).

• ClaudeBot — Anthropic, training crawler
• Claude-User — Anthropic, user-initiated fetcher
• Claude-SearchBot — Anthropic, search-index crawler
• GPTBot — OpenAI, training crawler
• OAI-SearchBot — OpenAI, search-index crawler
• ChatGPT-User — OpenAI, user-initiated fetcher
• PerplexityBot — Perplexity, search-index crawler
• Perplexity-User — Perplexity, user-initiated fetcher
• Google-Extended — Google, training crawler
• Google-Agent — Google, user-initiated fetcher
• Meta-ExternalAgent — Meta, training crawler

Scan your site to see how it treats these agents, or read the robots.txt guide.