Crystal Ball checks covered

ROBOTS_BLOCKED, ROBOTS_UNKNOWN, ROBOTS_TXT_FETCH_ERROR, NOINDEX_FOUND, NON_200_STATUS, ROUTE_STATUS_UNSTABLE

Guide detail

Choose Your Technical Level

This setting applies to the meaning, causes, and fix sections. Your choice carries to other Fix Library guides on this device.

Practical explanations and page-specific steps. Recommended for most readers.

What the flag means

Crystal Ball found evidence that one or more search or answer-system crawlers may not be able to retrieve the page normally. This is an eligibility warning, not a promise that allowing access will produce a citation.

  • OAI-SearchBot supports ChatGPT search visibility; GPTBot is a separate training crawler.
  • PerplexityBot supports Perplexity search visibility; Perplexity-User handles some user-triggered fetches.
  • Googlebot and Bingbot remain the important crawlers for their search indexes and AI experiences built on those indexes.
  • A successful request needs more than a friendly robots.txt rule: the final page must also survive CDN, WAF, authentication, and application checks.

Common causes

The usual culprits are broad robots.txt blocks, accidental noindex or nosnippet controls, 401 or 403 responses, bot-management rules, JavaScript challenges, CAPTCHAs, login walls, geo restrictions, and allowlists that trust a spoofable user-agent string.

  • Robots.txt controls cooperative crawling but does not grant access through a firewall.
  • Noindex affects indexing; nosnippet and max-snippet affect how much page text may be shown or reused in supported search previews.
  • Noarchive is not a universal citation control and is unsupported or interpreted differently across systems.
  • A browser returning 200 does not prove an automated client receives the same response.

How to fix it

Decide which uses you intend to allow, then make the narrowest change at every layer that owns access. Keep search and citation controls separate from model-training controls.

  • Allow OAI-SearchBot if ChatGPT search visibility is desired; control GPTBot separately for OpenAI training preferences.
  • Allow PerplexityBot if Perplexity search visibility is desired.
  • Keep Googlebot and Bingbot able to crawl indexable public pages and their required assets.
  • Test the exact URL with crawler-like GET requests and inspect origin, CDN, WAF, and application logs for 403, 429, challenge, or redirect responses.
  • Verify official bots with provider-published IP ranges or forward-confirmed reverse DNS where supported. Never trust the user-agent alone.

How to verify access safely

Verification should prove that an official crawler can fetch the canonical public URL without weakening protection for arbitrary automated traffic.

  • Use OpenAI and Perplexity's current published IP JSON endpoints for their bots.
  • Use Google's published crawler ranges or forward-confirmed reverse DNS for Googlebot.
  • Use Bing Webmaster Tools or forward-confirmed reverse DNS for Bingbot.
  • Recheck after rule propagation and monitor logs for the expected verified source and final 200 response.

Official references

FAQ

Does allowing an AI crawler guarantee citations?

No. Access is only an eligibility condition. Relevance, indexing, content quality, provider behavior, and the question still determine whether a page is used or cited.

Should I allow a bot by user-agent alone?

No. User-agent strings are easy to spoof. Combine the declared agent with current provider-published IP ranges or verified DNS methods where the provider supports them.

Is blocking model training the same as blocking search citations?

No. OpenAI documents OAI-SearchBot and GPTBot as independent controls, and Perplexity says PerplexityBot is for search rather than foundation-model training.

Need a Second Set of Eyes?

Run a scan, compare the finding with the right guide, and ask for support before changing risky SEO or server settings.

Contact Your SEO Wizard