Meta-ExternalFetcher user agent and robots.txt rules

Meta-ExternalFetcher fetches individual links at a user’s request, for example to help Meta AI complete a task on a website.

User-triggered May skip robots.txt Meta

User agent

As documented by Meta. Version numbers change; match on the token.

User agent
meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

robots.txt rules for Meta-ExternalFetcher

Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read.

Block Meta-ExternalFetcher

Disallows every path for Meta-ExternalFetcher. That asks Meta not to fetch your pages when its users ask, though Meta says user-triggered fetches may not always follow robots.txt.

robots.txt
User-agent: meta-externalfetcher
Disallow: /

Allow Meta-ExternalFetcher

Lets Meta-ExternalFetcher fetch every path, whatever “User-agent: *” says.

robots.txt
User-agent: meta-externalfetcher
Allow: /

Good to know

Meta says it may bypass robots.txt, because the fetch was requested by a user.

Meta-ExternalFetcher FAQ

What is Meta-ExternalFetcher?

Meta-ExternalFetcher fetches individual links at a user’s request, for example to help Meta AI complete a task on a website.

How do I block Meta-ExternalFetcher?

Add “User-agent: meta-externalfetcher” and “Disallow: /” to your robots.txt. That asks Meta not to fetch your pages when its users ask, though Meta says user-triggered fetches may not always follow robots.txt.

Does Meta-ExternalFetcher respect robots.txt?

Meta says that because a person starts these requests, robots.txt rules may not apply to Meta-ExternalFetcher.

What is the Meta-ExternalFetcher user agent?

Meta documents it as “meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)”. Version numbers change, so match on “meta-externalfetcher”.

How can I verify Meta-ExternalFetcher requests?

Meta publishes no IP list for Meta-ExternalFetcher. Treat the user agent as a hint, not proof.

Source: developers.facebook.com, checked Nov 17, 2026. AI Crawler Directory

Files are the start. Answers are the result.

Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.