Skip to main content
All Guides

Agent Access (Cloaking Detection)

Informational in 3.x — runs and reports but does not affect your AX score yet (it will gain weight in v4.0). This check probes your homepage with the user-agents of the 8 core AI crawlers and compares each response against the regular baseline. It catches the failure mode that is invisible to you but fatal for agents: robots.txt says Allow while your WAF returns 403.

Crawler allowed in robots.txt but blocked

A request carrying this crawler's user-agent token received an error (typically 403) while a regular request succeeds. Your declared policy and your enforcement disagree — agents trust robots.txt, attempt the crawl, and silently fail. Common causes:

  • Cloudflare "Block AI Crawlers" / AI Crawl Control: the one-toggle block rejects AI crawler UAs regardless of your robots.txt. Review it under Security → Bots, and allow the crawlers your robots.txt permits.
  • WAF managed rules: several vendors ship "AI scraper" rule sets matching UA tokens (GPTBot, ClaudeBot, CCBot, …). Audit custom and managed rules for UA-based matches.
  • Rate limiting / bot scoring: aggressive bot scores can challenge or drop crawler traffic. Check your bot-management logs for the specific token.

Reproduce manually:

curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0)" https://your-site.com
# Compare with:
curl -sI https://your-site.com

If you intend to block a crawler, declare it: add an explicit Disallow: / for it in robots.txt. ax-audit treats a server block that matches a robots.txt Disallow as consistent and does not penalize it.

Verified-bots caveat
ax-audit's probe is not the real crawler. If your WAF verifies bots cryptographically (Web Bot Auth) or by IP range, the genuine GPTBot may pass while this unverified probe is rejected. Before changing rules, confirm against your WAF logs whether the real crawler traffic is being served.

Crawler receives reduced content

The server returned 200 to the crawler UA, but with less than half the visible text of the baseline response. The crawler is getting an interstitial, a JS challenge page ("Checking your browser…"), or conditionally rendered content — and that is what gets indexed.

  • Check whether your CDN serves challenge pages with status 200 to suspected bots (managed challenges often do).
  • If you conditionally render for bots, make sure the bot variant contains the full main content — serving more (e.g., pre-rendered HTML for an SPA) is fine; serving less defeats the crawl.
  • Compare the actual payloads: curl -A "...GPTBot..." https://your-site.com | wc -c vs the default fetch.

Baseline request failed

The default-UA request to your homepage failed, so there is nothing to compare crawler responses against. Fix general reachability first — see the other failing checks in your report — and re-run the audit.