GPTBot: OpenAI's crawler explained

GPTBot is the OpenAI crawler that collects content which may be used to train OpenAI's models. It is separate from OAI-SearchBot, which surfaces sites in ChatGPT's search features, and from ChatGPT-User, which acts on a user's request. You can allow or block each one in robots.txt.

By the AnswerHawk team Updated November 17, 2026 1 min read

GPTBot is the web crawler OpenAI uses to gather content that may be used to train its generative AI models (OpenAI). It is one of three OpenAI user agents, and each one needs its own decision in robots.txt. Check whether GPTBot can crawl your site before you change any rule.

GPTBot, OAI-SearchBot and ChatGPT-User

User agent What OpenAI uses it for robots.txt IP ranges
GPTBot Crawls content that may be used to train OpenAI's foundation models Honoured gptbot.json
OAI-SearchBot Surfaces websites in ChatGPT's search features Honoured searchbot.json
ChatGPT-User Certain actions a user starts in ChatGPT and custom GPTs; not automatic crawling "robots.txt rules may not apply" chatgpt-user.json

Source: OpenAI, Overview of OpenAI crawlers.

robots.txt rules for GPTBot

Block training, allow ChatGPT search:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Allow everything from OpenAI:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

Block GPTBot from one folder only:

User-agent: GPTBot
Disallow: /private/

Rules apply per user agent, so a Disallow for GPTBot doesn't affect OAI-SearchBot. Remember that a CDN or firewall "block AI bots" switch can override what your robots.txt says.

How to verify GPTBot traffic

A user agent string is easy to fake. OpenAI publishes the IP ranges of each crawler as JSON (linked in the table above). Match the IP of a request against the right list before you trust it, and review your logs after any rule change.

Does GPTBot affect visibility in ChatGPT?

Visibility in ChatGPT's search answers depends on the search crawler, not on GPTBot: OpenAI describes OAI-SearchBot as the agent used to surface websites in ChatGPT's search features (OpenAI). Answers that come from the model alone reflect what the model learned in training, which is where GPTBot's role sits. If AI visibility matters to you, measure it on a fixed set of prompts before and after changing these rules.

Sources

  1. Overview of OpenAI crawlers · OpenAI
  2. GPTBot IP ranges · OpenAI
  3. OAI-SearchBot IP ranges · OpenAI
  4. ChatGPT-User IP ranges · OpenAI

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT search?

GPTBot is the training crawler. OpenAI uses a different crawler, OAI-SearchBot, to surface websites in ChatGPT's search features, so the two are controlled separately in robots.txt.

What is the GPTBot user agent token?

GPTBot. Use it as the User-agent line in robots.txt.

Does ChatGPT-User follow robots.txt?

OpenAI says ChatGPT-User acts on user-initiated requests and that robots.txt rules may not apply to it.

How can I verify a request really comes from GPTBot?

OpenAI publishes the IP ranges of each crawler as JSON files. Compare the request's IP with the published list instead of trusting the user agent alone.

Should I block GPTBot?

It's a choice about training, not about search visibility. Many sites block training and allow search crawlers. Decide per crawler, and check your CDN rules match your robots.txt.

See where you stand in AI answers.

Free for 7 days. Your first answers in minutes.