CCBot user agent and robots.txt rules
CCBot is Common Crawl’s crawler. It builds an open archive of the web that is free to download and widely used as AI training data.
User agent
As documented by Common Crawl. Version numbers change; match on the token.
CCBot/2.0 (https://commoncrawl.org/faq/)
robots.txt rules for CCBot
Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read.
Block CCBot
Disallows every path for CCBot. That keeps your future pages out of what Common Crawl may use for training. It does not change search results.
User-agent: CCBot Disallow: /
Allow CCBot
Lets CCBot fetch every path, whatever “User-agent: *” says.
User-agent: CCBot Allow: /
Good to know
Others sometimes fake its user agent. Common Crawl publishes its IP ranges and supports reverse DNS under crawl.commoncrawl.org.
CCBot FAQ
What is CCBot?
CCBot is Common Crawl’s crawler. It builds an open archive of the web that is free to download and widely used as AI training data.
How do I block CCBot?
Add “User-agent: CCBot” and “Disallow: /” to your robots.txt. That keeps your future pages out of what Common Crawl may use for training. It does not change search results.
Does CCBot respect robots.txt?
Yes. Common Crawl says CCBot follows robots.txt rules written for it.
What is the CCBot user agent?
Common Crawl documents it as “CCBot/2.0 (https://commoncrawl.org/faq/)”. Version numbers change, so match on “CCBot”.
How can I verify CCBot requests?
Match the request IP against Common Crawl’s published list (https://index.commoncrawl.org/ccbot.json). The user agent alone can be faked.
Source: commoncrawl.org, checked Nov 17, 2026. AI Crawler Directory
Files are the start. Answers are the result.
Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.