Claude-User user agent and robots.txt rules
Claude-User fetches a page when a Claude user asks a question that needs it. Blocking it stops Claude from retrieving your content for those questions.
User agent
Anthropic documents the name only: requests contain “Claude-User”.
robots.txt rules for Claude-User
Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read.
Block Claude-User
Disallows every path for Claude-User. That asks Anthropic not to fetch your pages when its users ask, though Anthropic says user-triggered fetches may not always follow robots.txt.
User-agent: Claude-User Disallow: /
Allow Claude-User
Lets Claude-User fetch every path, whatever “User-agent: *” says.
User-agent: Claude-User Allow: /
Good to know
Anthropic says its bots honour robots.txt, including this one.
Claude-User FAQ
What is Claude-User?
Claude-User fetches a page when a Claude user asks a question that needs it. Blocking it stops Claude from retrieving your content for those questions.
How do I block Claude-User?
Add “User-agent: Claude-User” and “Disallow: /” to your robots.txt. That asks Anthropic not to fetch your pages when its users ask, though Anthropic says user-triggered fetches may not always follow robots.txt.
Does Claude-User respect robots.txt?
Yes. Anthropic says Claude-User follows robots.txt rules written for it.
What is the Claude-User user agent?
Anthropic documents the name “Claude-User”, which appears in its requests.
How can I verify Claude-User requests?
Match the request IP against Anthropic’s published list (https://claude.com/crawling/bots.json). The user agent alone can be faked.
Source: support.claude.com, checked Nov 17, 2026. AI Crawler Directory
Files are the start. Answers are the result.
Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.