Google-Agent user agent and robots.txt rules

Google-Agent is used by agents on Google infrastructure that browse the web and act on a user’s request.

User-triggered May skip robots.txt Google

User agent

Google documents the name only: requests contain “Google-Agent”.

robots.txt rules for Google-Agent

Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read.

Block Google-Agent

Disallows every path for Google-Agent. That asks Google not to fetch your pages when its users ask, though Google says user-triggered fetches may not always follow robots.txt.

robots.txt
User-agent: Google-Agent
Disallow: /

Allow Google-Agent

Lets Google-Agent fetch every path, whatever “User-agent: *” says.

robots.txt
User-agent: Google-Agent
Allow: /

Good to know

Google says user-triggered fetchers generally ignore robots.txt, because a person asked for the page.

Google-Agent FAQ

What is Google-Agent?

Google-Agent is used by agents on Google infrastructure that browse the web and act on a user’s request.

How do I block Google-Agent?

Add “User-agent: Google-Agent” and “Disallow: /” to your robots.txt. That asks Google not to fetch your pages when its users ask, though Google says user-triggered fetches may not always follow robots.txt.

Does Google-Agent respect robots.txt?

Google says that because a person starts these requests, robots.txt rules may not apply to Google-Agent.

What is the Google-Agent user agent?

Google documents the name “Google-Agent”, which appears in its requests.

How can I verify Google-Agent requests?

Match the request IP against Google’s published list (https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json). The user agent alone can be faked.

Source: developers.google.com, checked Nov 17, 2026. AI Crawler Directory

Files are the start. Answers are the result.

Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.