ClaudeBot

Anthropic · Training crawler

ClaudeBot is Anthropic’s training crawler — it crawls the web to collect content for model training. Blocking it opts your content out of future models but does not affect live answers.

User-agent string

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

Source: community-documented (Anthropic describes the bot but does not publish the full string).

Does ClaudeBot respect robots.txt?

Respects robots.txt, per Anthropic’s crawler documentation.

Allow or block ClaudeBot

# Allow (recommended if you want ClaudeBot traffic)
User-agent: ClaudeBot
Allow: /

# Block
User-agent: ClaudeBot
Disallow: /

Verifying it’s really ClaudeBot

Anthropic publishes crawler IP ranges at claude.com/crawling/bots.json (216.73.216.0/22 plus individual addresses). A ClaudeBot user-agent from an IP outside those ranges is an impersonator. User-agent strings can be spoofed by anyone — identity claims are only trustworthy when the source IP matches the operator’s published ranges.

What site owners should know

ClaudeBot collects web content for training Anthropic’s models. It is distinct from Claude-User (live fetches for users) and Claude-SearchBot (search indexing) — blocking ClaudeBot does not block the other two.

Scan your site to see how your robots.txt, content negotiation, and discovery files treat AI agents — including whether you’re blocking agents you meant to allow. Background: the robots.txt guide and agent readiness.

All AI agent user-agents

ClaudeBot · Claude-User · Claude-SearchBot · GPTBot · OAI-SearchBot · ChatGPT-User · PerplexityBot · Perplexity-User · Google-Extended · Google-Agent · Meta-ExternalAgent