Meta-ExternalFetcher

Meta · User-initiated fetcher · Updated 2026-07-19

Meta-ExternalFetcher is Meta’s user-initiated fetcher — it fetches a page live because a human asked the assistant about it right now. Most vendors state this tier ignores robots.txt, reasoning that a human requested the specific page.

User-agent string

meta-externalfetcher/1.1

Source: vendor-documented at developers.facebook.com (web crawlers page).

Does Meta-ExternalFetcher respect robots.txt?

Meta’s exact wording: “The Meta-ExternalFetcher crawler may bypass robots.txt because it performs fetches that were requested by the user.”

Allow or block Meta-ExternalFetcher

# Allow (recommended if you want Meta-ExternalFetcher traffic)
User-agent: meta-externalfetcher
Allow: /

# Block
User-agent: meta-externalfetcher
Disallow: /

Advisory only for this tier: Meta’s documentation says this fetcher may not honor robots.txt (see above). For a firm block, filter by verified IP ranges or use a WAF rule.

Verifying it’s really Meta-ExternalFetcher

Meta’s crawler page suggests allowlisting by user-agent or IP but publishes no IP list for this fetcher. User-agent strings can be spoofed by anyone — identity claims are only trustworthy when the source IP matches the operator’s published ranges.

What site owners should know

Fetches individual links at a user’s request and supports Meta’s agentic AI features — including helping AI navigate websites to complete tasks. Blocking meta-externalagent (the training crawler) does not stop this fetcher, and robots.txt may not either.

Scan your site to see how your robots.txt, content negotiation, and discovery files treat AI agents — including whether you’re blocking agents you meant to allow. Background: the robots.txt guide, how AI agents browse, and agent readiness.

All AI agent user-agents

ClaudeBot · Claude-User · Claude-SearchBot · GPTBot · OAI-SearchBot · ChatGPT-User · PerplexityBot · Perplexity-User · Google-Extended · Google-Agent · Google-CloudVertexBot · Meta-ExternalAgent · Meta-ExternalFetcher · MistralAI-User · Applebot · Applebot-Extended · Amazonbot · bingbot · DuckAssistBot · CCBot · Bytespider