The AthenaHQ crawler
AthenaHQ reads public pages on websites our customers have asked us to audit. We check what AI assistants can see when they answer questions about a brand: whether a page loads without JavaScript, what its title and structured data say, and whether AI crawlers are allowed to read it.
How to identify us
Our audit requests carry this user agent:
AthenaHQ-Bot/1.0 (+https://app.athenahq.ai/bot)
The requests that measure your rules are the exception. To test whether your rules admit AI crawlers, we send a small number to your homepage carrying those crawlers’ own user agents (GPTBot, ClaudeBot, PerplexityBot and similar), and a few as an ordinary browser to compare against. Those are the measurement, so they cannot announce themselves as us.
We do not publish fixed IP addresses. A rule can match the user agent above, but anyone can copy a user agent, so treat that as a convenience rather than proof of identity.
Allowing us with a crawler key
For a rule only AthenaHQ can meet, an admin of your organization’s AthenaHQ account can create a crawler key under Settings, General, Crawler access. We then send it with the audit’s requests to your own websites, in this header:
x-athenahq-crawler-key: <your crawler key>
In Cloudflare, for example, a WAF custom rule with the expression below and the Skip action lets the audit past your bot checks:
any(http.request.headers["x-athenahq-crawler-key"][*] eq "<your crawler key>")
The key goes only to the websites in that AthenaHQ account, only over HTTPS, never to any other site, and not on the few requests that measure your rules under other crawlers’ names. Rotating it in AthenaHQ replaces it within a few minutes; update the rule when you do.
If your firewall turns us away
Bot protection often refuses an unrecognized crawler before returning any page. When a page, a link check, a sitemap or robots.txt comes back refused, we retry that request once as a browser so the audit can still finish, using:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
We would rather not do that. Allowing the AthenaHQ user agent keeps our traffic identifiable in your logs, and it lets us report your crawler rules accurately instead of measuring what we could reach around them.
What we do and do not do
- We request public pages, on the scheme the site serves them on. Site audits read a sample of pages; full inventory audits can read thousands of pages and check their links. The total depends on the crawl scope selected in Site Health.
- We read and cache robots.txt before crawling. We respect the Crawl-delay for AthenaHQ-Bot, or the wildcard group when no AthenaHQ-Bot group exists, across our crawler requests to the same host, including retries and redirects. Site Health shows a detected delay, and audits can take longer when one is set.
- We also read robots.txt to find your sitemaps and to report your crawler rules back to you. We do not treat a Disallow for our own user agent as a stop: the audit runs because the site’s owner asked for it through an AthenaHQ account. To stop it, ask that account holder, or write to us, or block the user agent at your edge.
- We do not submit forms, sign in, or attempt to reach anything behind authentication.
- We do not collect personal data from the pages we read.
Questions about a request you have seen from us: support@athenahq.ai.