ClaudeBot user agent and robots.txt rules
ClaudeBot is Anthropic’s crawler for web content that may be used to train its models. Blocking it keeps your future content out of Anthropic’s training data.
User agent
Anthropic documents the name only: requests contain “ClaudeBot”.
robots.txt rules for ClaudeBot
Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read.
Block ClaudeBot
Disallows every path for ClaudeBot. That keeps your future pages out of what Anthropic may use for training. It does not change search results.
User-agent: ClaudeBot Disallow: /
Allow ClaudeBot
Lets ClaudeBot fetch every path, whatever “User-agent: *” says.
User-agent: ClaudeBot Allow: /
Good to know
ClaudeBot also honours the non-standard Crawl-delay directive. Anthropic advises against IP blocking to opt out, since it can stop the bot from reading your robots.txt.
ClaudeBot FAQ
What is ClaudeBot?
ClaudeBot is Anthropic’s crawler for web content that may be used to train its models. Blocking it keeps your future content out of Anthropic’s training data.
How do I block ClaudeBot?
Add “User-agent: ClaudeBot” and “Disallow: /” to your robots.txt. That keeps your future pages out of what Anthropic may use for training. It does not change search results.
Does ClaudeBot respect robots.txt?
Yes. Anthropic says ClaudeBot follows robots.txt rules written for it.
What is the ClaudeBot user agent?
Anthropic documents the name “ClaudeBot”, which appears in its requests.
How can I verify ClaudeBot requests?
Match the request IP against Anthropic’s published list (https://claude.com/crawling/bots.json). The user agent alone can be faked.
Source: support.claude.com, checked Nov 17, 2026. AI Crawler Directory
Files are the start. Answers are the result.
Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.