Meta-ExternalAgent

Meta · Training crawler

Meta-ExternalAgent is Meta’s training crawler — it crawls the web to collect content for model training. Blocking it opts your content out of future models but does not affect live answers.

User-agent string

meta-externalagent/1.1

Source: vendor-documented at developers.facebook.com (web crawlers page); a longer form with a docs URL appears in server logs.

Does Meta-ExternalAgent respect robots.txt?

Respects robots.txt.

Allow or block Meta-ExternalAgent

# Allow (recommended if you want Meta-ExternalAgent traffic)
User-agent: meta-externalagent
Allow: /

# Block
User-agent: meta-externalagent
Disallow: /

Verifying it’s really Meta-ExternalAgent

Documented on Meta’s webmaster crawler pages. User-agent strings can be spoofed by anyone — identity claims are only trustworthy when the source IP matches the operator’s published ranges.

What site owners should know

Meta’s AI training crawler. Its sibling Meta-ExternalFetcher handles user-initiated link fetches for Meta AI and may bypass robots.txt for user-provided URLs.

Scan your site to see how your robots.txt, content negotiation, and discovery files treat AI agents — including whether you’re blocking agents you meant to allow. Background: the robots.txt guide and agent readiness.

All AI agent user-agents

ClaudeBot · Claude-User · Claude-SearchBot · GPTBot · OAI-SearchBot · ChatGPT-User · PerplexityBot · Perplexity-User · Google-Extended · Google-Agent · Meta-ExternalAgent