AnswerHawkBot, the AnswerHawk crawler

A small, polite fetcher. It reads a few public files and the pages AI engines cite, never runs JavaScript, never signs in and never feeds AI training.

User agent AnswerHawkBot/1.0 (+https://answerhawk.io/bot)

Last updated November 17, 2026

What AnswerHawkBot is

AnswerHawk tracks how AI engines answer questions about brands. Answers come from the engines through data vendors and official APIs; AnswerHawkBot is not used for that. It only fetches a few things from websites, and every request carries the user agent above.

When it visits and what it reads

Request When What it reads robots.txt Rate limit
Cited page check After an AI answer cites a page, at most once a week per page and brand The cited page, to see if it lists the tracked brand Followed 10 / min per site
Favicon For domains that appear in a project, every 30 days The home page <head> and /favicon.ico Standard file 6 / min per site
Site details When someone enters your domain in onboarding or the free AI Visibility Checker The home page title and description On request One request each time
Free tools When someone runs the llms.txt Checker, llms.txt Generator or AI Crawler Checker on your domain robots.txt, llms.txt, llms-full.txt, sitemaps and a few pages On request 20 / min per site
llms.txt directory Sites listed in the directory, every 30 days robots.txt, then /llms.txt Followed 20 / min per site

How it behaves

  • only GET requests for public URLs; it never submits forms, signs in, keeps cookies or runs JavaScript;
  • explicit timeouts (8 seconds per request), at most 4 redirects, and a size cap on every response;
  • per-site rate limits (see the table), so a site never receives bursts from us;
  • only public internet addresses: private, local and reserved addresses are refused on every redirect;
  • results are cached: a cited page is checked at most once a week for each brand.

What we keep

  • cited pages: only the yes/no answer to "does this page list the brand?", for 7 days. Page content is not stored;
  • favicons: the icon file, refreshed every 30 days, shown next to your domain in the app;
  • llms.txt directory: a copy of public llms.txt files and their history, shown on the directory page;
  • nothing we fetch is sold or used to train AI models.

Control access with robots.txt

The cited-page check and the llms.txt directory follow robots.txt. They read the group for AnswerHawkBot (or * when there is none). To stop them on your whole site:

User-agent: AnswerHawkBot
Disallow: /

To block only part of the site:

User-agent: AnswerHawkBot
Disallow: /private/

Favicons are standard public files, and the free tools read robots.txt, llms.txt and sitemaps only when someone asks for them. If you want us to stop every request to your domain, contact us and we will exclude it.

Verify a request

Real requests send exactly the user agent above and come from AnswerHawk servers. We do not publish a fixed list of IP addresses yet. If you see traffic that uses our name but ignores these rules, tell us with the time, URL and IP address.

Looking for the crawlers of AI companies (GPTBot, ClaudeBot, PerplexityBot…)? See the AI crawler directory or test your site with the AI Crawler Checker.

AnswerHawkBot FAQ

Why is AnswerHawkBot visiting my site?

Most often because an AI engine cited one of your pages in an answer we track, and we check whether that page lists the customer's brand. It may also fetch your favicon or, when someone asks for it in a free tool, your robots.txt or llms.txt.

Does AnswerHawkBot collect content to train AI?

No. We do not train AI models and we do not sell what we fetch. For cited pages we keep only a yes or no answer for 7 days.

How do I block AnswerHawkBot?

Add "User-agent: AnswerHawkBot" with "Disallow: /" to your robots.txt. To stop every request to your domain, including favicons, contact us and we will exclude it.

Does AnswerHawkBot run JavaScript or fill in forms?

No. It makes plain GET requests, does not keep cookies, does not sign in and does not submit anything.

Is AnswerHawkBot the same as GPTBot or ClaudeBot?

No. Those crawlers belong to AI companies. AnswerHawkBot belongs to AnswerHawk, an AI visibility tracker, and is not used to collect AI answers.