Googlebot user agent and robots.txt rules
Googlebot is Google Search’s crawler. Google’s AI Overviews and AI Mode are Search features, so Googlebot’s robots.txt rules are the control for them too.
User agent
As documented by Google. Version numbers change; match on the token.
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
robots.txt rules for Googlebot
Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read.
Block Googlebot
Disallows every path for Googlebot. That keeps your pages out of Google’s AI search index, so its answers can’t cite them.
User-agent: Googlebot Disallow: /
Allow Googlebot
Lets Googlebot fetch every path, whatever “User-agent: *” says.
User-agent: Googlebot Allow: /
Good to know
Blocking Googlebot also removes you from regular Google Search. To limit what AI features show, Google points to nosnippet, data-nosnippet, max-snippet or noindex instead.
Other Google crawlers
Google-Extended
Google-Extended is a robots.txt token, not a crawler. It controls whether pages Google already…
OtherGoogle-CloudVertexBot
Google-CloudVertexBot runs crawls that site owners request to build Vertex AI Agents. Google…
User-triggeredGoogle-GeminiNotebook
Google-GeminiNotebook fetches the URLs that people add as sources to Gemini Notebook projects.…
User-triggeredGoogle-Agent
Google-Agent is used by agents on Google infrastructure that browse the web and act on a…
Googlebot FAQ
What is Googlebot?
Googlebot is Google Search’s crawler. Google’s AI Overviews and AI Mode are Search features, so Googlebot’s robots.txt rules are the control for them too.
How do I block Googlebot?
Add “User-agent: Googlebot” and “Disallow: /” to your robots.txt. That keeps your pages out of Google’s AI search index, so its answers can’t cite them.
Does Googlebot respect robots.txt?
Yes. Google says Googlebot follows robots.txt rules written for it.
What is the Googlebot user agent?
Google documents it as “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”. Version numbers change, so match on “Googlebot”.
How can I verify Googlebot requests?
Match the request IP against Google’s published list (https://developers.google.com/static/search/apis/ipranges/googlebot.json). The user agent alone can be faked.
Source: developers.google.com, checked Nov 17, 2026. AI Crawler Directory
Files are the start. Answers are the result.
Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.