GPTBot user agent and robots.txt rules
GPTBot is OpenAI’s crawler for content that may be used to train its generative AI foundation models. Disallowing it tells OpenAI not to use your site for training.
User agent
As documented by OpenAI. Version numbers change; match on the token.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
robots.txt rules for GPTBot
Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read.
Block GPTBot
Disallows every path for GPTBot. That keeps your future pages out of what OpenAI may use for training. It does not change search results.
User-agent: GPTBot Disallow: /
Allow GPTBot
Lets GPTBot fetch every path, whatever “User-agent: *” says.
User-agent: GPTBot Allow: /
Good to know
GPTBot is not the crawler behind ChatGPT search: that is OAI-SearchBot. You can block one and allow the other.
Other OpenAI crawlers
OAI-SearchBot
OAI-SearchBot is OpenAI’s crawler for surfacing websites in ChatGPT’s search features. Sites…
User-triggeredChatGPT-User
ChatGPT-User fetches pages for actions people start in ChatGPT and custom GPTs, such as…
OtherOAI-AdsBot
OAI-AdsBot checks the landing pages of ads submitted to ChatGPT, for safety and to decide when…
GPTBot FAQ
What is GPTBot?
GPTBot is OpenAI’s crawler for content that may be used to train its generative AI foundation models. Disallowing it tells OpenAI not to use your site for training.
How do I block GPTBot?
Add “User-agent: GPTBot” and “Disallow: /” to your robots.txt. That keeps your future pages out of what OpenAI may use for training. It does not change search results.
Does GPTBot respect robots.txt?
Yes. OpenAI says GPTBot follows robots.txt rules written for it.
What is the GPTBot user agent?
OpenAI documents it as “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot”. Version numbers change, so match on “GPTBot”.
How can I verify GPTBot requests?
Match the request IP against OpenAI’s published list (https://openai.com/gptbot.json). The user agent alone can be faked.
Source: developers.openai.com, checked Nov 17, 2026. AI Crawler Directory
Files are the start. Answers are the result.
Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.