Google-Agent user agent and robots.txt rules
Google-Agent is used by agents on Google infrastructure that browse the web and act on a user’s request.
User agent
Google documents the name only: requests contain “Google-Agent”.
robots.txt rules for Google-Agent
Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read.
Block Google-Agent
Disallows every path for Google-Agent. That asks Google not to fetch your pages when its users ask, though Google says user-triggered fetches may not always follow robots.txt.
User-agent: Google-Agent Disallow: /
Allow Google-Agent
Lets Google-Agent fetch every path, whatever “User-agent: *” says.
User-agent: Google-Agent Allow: /
Good to know
Google says user-triggered fetchers generally ignore robots.txt, because a person asked for the page.
Other Google crawlers
Googlebot
Googlebot is Google Search’s crawler. Google’s AI Overviews and AI Mode are Search features,…
AI trainingGoogle-Extended
Google-Extended is a robots.txt token, not a crawler. It controls whether pages Google already…
OtherGoogle-CloudVertexBot
Google-CloudVertexBot runs crawls that site owners request to build Vertex AI Agents. Google…
User-triggeredGoogle-GeminiNotebook
Google-GeminiNotebook fetches the URLs that people add as sources to Gemini Notebook projects.…
Google-Agent FAQ
What is Google-Agent?
Google-Agent is used by agents on Google infrastructure that browse the web and act on a user’s request.
How do I block Google-Agent?
Add “User-agent: Google-Agent” and “Disallow: /” to your robots.txt. That asks Google not to fetch your pages when its users ask, though Google says user-triggered fetches may not always follow robots.txt.
Does Google-Agent respect robots.txt?
Google says that because a person starts these requests, robots.txt rules may not apply to Google-Agent.
What is the Google-Agent user agent?
Google documents the name “Google-Agent”, which appears in its requests.
How can I verify Google-Agent requests?
Match the request IP against Google’s published list (https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json). The user agent alone can be faked.
Source: developers.google.com, checked Nov 17, 2026. AI Crawler Directory
Files are the start. Answers are the result.
Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.