Meta-WebIndexer user agent and robots.txt rules

Meta-WebIndexer analyses online content to improve the quality of Meta AI search results. Allowing it lets Meta AI reference your pages in its answers.

AI search Follows robots.txt Meta

User agent

As documented by Meta. Version numbers change; match on the token.

User agent
meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

robots.txt rules for Meta-WebIndexer

Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read.

Block Meta-WebIndexer

Disallows every path for Meta-WebIndexer. That keeps your pages out of Meta’s AI search index, so its answers can’t cite them.

robots.txt
User-agent: meta-webindexer
Disallow: /

Allow Meta-WebIndexer

Lets Meta-WebIndexer fetch every path, whatever “User-agent: *” says.

robots.txt
User-agent: meta-webindexer
Allow: /

Good to know

Meta says crawlers may cache your robots.txt for up to 24 hours.

Meta-WebIndexer FAQ

What is Meta-WebIndexer?

Meta-WebIndexer analyses online content to improve the quality of Meta AI search results. Allowing it lets Meta AI reference your pages in its answers.

How do I block Meta-WebIndexer?

Add “User-agent: meta-webindexer” and “Disallow: /” to your robots.txt. That keeps your pages out of Meta’s AI search index, so its answers can’t cite them.

Does Meta-WebIndexer respect robots.txt?

Yes. Meta says Meta-WebIndexer follows robots.txt rules written for it.

What is the Meta-WebIndexer user agent?

Meta documents it as “meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)”. Version numbers change, so match on “meta-webindexer”.

How can I verify Meta-WebIndexer requests?

Meta publishes no IP list for Meta-WebIndexer. Treat the user agent as a hint, not proof.

Source: developers.facebook.com, checked Nov 17, 2026. AI Crawler Directory

Files are the start. Answers are the result.

Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.