AgentGrade

Claude-User

Anthropic · User-initiated fetcher

Claude-User is Anthropic’s user-initiated fetcher — it fetches a page live because a human asked the assistant about it right now. Most vendors state this tier ignores robots.txt, reasoning that a human requested the specific page.

User-agent string

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)

Source: community-documented (Anthropic describes the bot but does not publish the full string).

Does Claude-User respect robots.txt?

Anthropic states Claude-User respects robots.txt — the documented exception in this tier: OpenAI, Perplexity, Google, and Meta all say their user-initiated fetchers may bypass or generally ignore it.

Allow or block Claude-User

# Allow (recommended if you want Claude-User traffic)
User-agent: Claude-User
Allow: /

# Block
User-agent: Claude-User
Disallow: /

Advisory only for this tier: Anthropic’s documentation says this fetcher may not honor robots.txt (see above). For a firm block, filter by verified IP ranges or use a WAF rule.

Verifying it’s really Claude-User

Same published IP ranges as all Anthropic crawlers: claude.com/crawling/bots.json. User-agent strings can be spoofed by anyone — identity claims are only trustworthy when the source IP matches the operator’s published ranges.

What site owners should know

Claude-User fires when a person asks Claude about your site in claude.ai chat and other Anthropic-hosted surfaces. Note: Claude Code’s WebFetch runs from the user’s own machine, so those visits arrive with a different UA from a residential IP, not from Anthropic’s ranges. Blocking Claude-User makes your site invisible to Claude users at the moment they are asking about you.

Scan your site to see how your robots.txt, content negotiation, and discovery files treat AI agents — including whether you’re blocking agents you meant to allow. Background: the robots.txt guide and agent readiness.

All AI agent user-agents

ClaudeBot · Claude-User · Claude-SearchBot · GPTBot · OAI-SearchBot · ChatGPT-User · PerplexityBot · Perplexity-User · Google-Extended · Google-Agent · Meta-ExternalAgent