Google-Extended

Google · Training crawler

Google-Extended is Google’s training crawler — it crawls the web to collect content for model training. Blocking it opts your content out of future models but does not affect live answers.

User-agent string

(none — Google-Extended is a robots.txt product token only; no separate crawler or user-agent string exists)

Source: vendor-documented at developers.google.com (Google crawler overview).

Does Google-Extended respect robots.txt?

It exists only as a robots.txt token: disallowing Google-Extended opts your content out of Gemini training and grounding without affecting Google Search indexing or ranking.

Allow or block Google-Extended

User-agent: Google-Extended
Disallow: /

Verifying it’s really Google-Extended

Not applicable — no requests carry this identity; regular Googlebot does the fetching. User-agent strings can be spoofed by anyone — identity claims are only trustworthy when the source IP matches the operator’s published ranges.

What site owners should know

The most misunderstood entry in the ecosystem: blocking Google-Extended does not block any traffic, and allowing it adds none. It is purely a policy switch read by Googlebot.

Scan your site to see how your robots.txt, content negotiation, and discovery files treat AI agents — including whether you’re blocking agents you meant to allow. Background: the robots.txt guide and agent readiness.

All AI agent user-agents

ClaudeBot · Claude-User · Claude-SearchBot · GPTBot · OAI-SearchBot · ChatGPT-User · PerplexityBot · Perplexity-User · Google-Extended · Google-Agent · Meta-ExternalAgent