Google-Agent

Google · User-initiated fetcher

Google-Agent is Google’s user-initiated fetcher — it fetches a page live because a human asked the assistant about it right now. Most vendors state this tier ignores robots.txt, reasoning that a human requested the specific page.

User-agent string

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36

Source: vendor-documented in Google’s user-triggered fetchers reference (a mobile variant also exists; W.X.Y.Z is the current Chrome version).

Does Google-Agent respect robots.txt?

Google’s user-triggered fetchers “generally ignore robots.txt rules” — a human requested the fetch.

Allow or block Google-Agent

# Allow (recommended if you want Google-Agent traffic)
User-agent: Google-Agent
Allow: /

# Block
User-agent: Google-Agent
Disallow: /

Advisory only for this tier: Google’s documentation says this fetcher may not honor robots.txt (see above). For a firm block, filter by verified IP ranges or use a WAF rule.

Verifying it’s really Google-Agent

Google documents its fetchers and IP ranges in the crawler infrastructure reference. User-agent strings can be spoofed by anyone — identity claims are only trustworthy when the source IP matches the operator’s published ranges.

What site owners should know

The fetcher behind Google-hosted agents navigating the web on a user’s behalf — the Gemini-era agentic traffic tier.

Scan your site to see how your robots.txt, content negotiation, and discovery files treat AI agents — including whether you’re blocking agents you meant to allow. Background: the robots.txt guide and agent readiness.

All AI agent user-agents

ClaudeBot · Claude-User · Claude-SearchBot · GPTBot · OAI-SearchBot · ChatGPT-User · PerplexityBot · Perplexity-User · Google-Extended · Google-Agent · Meta-ExternalAgent