GPTBot user agent and robots.txt rules

GPTBot is OpenAI’s crawler for content that may be used to train its generative AI foundation models. Disallowing it tells OpenAI not to use your site for training.

AI training Follows robots.txt OpenAI

User agent

As documented by OpenAI. Version numbers change; match on the token.

User agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

robots.txt rules for GPTBot

Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read.

Block GPTBot

Disallows every path for GPTBot. That keeps your future pages out of what OpenAI may use for training. It does not change search results.

robots.txt
User-agent: GPTBot
Disallow: /

Allow GPTBot

Lets GPTBot fetch every path, whatever “User-agent: *” says.

robots.txt
User-agent: GPTBot
Allow: /

Good to know

GPTBot is not the crawler behind ChatGPT search: that is OAI-SearchBot. You can block one and allow the other.

GPTBot FAQ

What is GPTBot?

GPTBot is OpenAI’s crawler for content that may be used to train its generative AI foundation models. Disallowing it tells OpenAI not to use your site for training.

How do I block GPTBot?

Add “User-agent: GPTBot” and “Disallow: /” to your robots.txt. That keeps your future pages out of what OpenAI may use for training. It does not change search results.

Does GPTBot respect robots.txt?

Yes. OpenAI says GPTBot follows robots.txt rules written for it.

What is the GPTBot user agent?

OpenAI documents it as “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot”. Version numbers change, so match on “GPTBot”.

How can I verify GPTBot requests?

Match the request IP against OpenAI’s published list (https://openai.com/gptbot.json). The user agent alone can be faked.

Source: developers.openai.com, checked Nov 17, 2026. AI Crawler Directory

Files are the start. Answers are the result.

Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.