Claude-SearchBot user agent and robots.txt rules
Claude-SearchBot crawls the web to improve the relevance and accuracy of Claude’s search answers. Blocking it keeps your pages out of that index.
User agent
Anthropic documents the name only: requests contain “Claude-SearchBot”.
robots.txt rules for Claude-SearchBot
Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read.
Block Claude-SearchBot
Disallows every path for Claude-SearchBot. That keeps your pages out of Anthropic’s AI search index, so its answers can’t cite them.
User-agent: Claude-SearchBot Disallow: /
Allow Claude-SearchBot
Lets Claude-SearchBot fetch every path, whatever “User-agent: *” says.
User-agent: Claude-SearchBot Allow: /
Good to know
Anthropic notes that blocking it may reduce your visibility in user search results.
Claude-SearchBot FAQ
What is Claude-SearchBot?
Claude-SearchBot crawls the web to improve the relevance and accuracy of Claude’s search answers. Blocking it keeps your pages out of that index.
How do I block Claude-SearchBot?
Add “User-agent: Claude-SearchBot” and “Disallow: /” to your robots.txt. That keeps your pages out of Anthropic’s AI search index, so its answers can’t cite them.
Does Claude-SearchBot respect robots.txt?
Yes. Anthropic says Claude-SearchBot follows robots.txt rules written for it.
What is the Claude-SearchBot user agent?
Anthropic documents the name “Claude-SearchBot”, which appears in its requests.
How can I verify Claude-SearchBot requests?
Match the request IP against Anthropic’s published list (https://claude.com/crawling/bots.json). The user agent alone can be faked.
Source: support.claude.com, checked Nov 17, 2026. AI Crawler Directory
Files are the start. Answers are the result.
Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.