Crystal Ball checks covered

ROBOTS_BLOCKED, ROBOTS_UNKNOWN, robots.txt warnings

Guide detail

Choose Your Technical Level

This setting applies to the meaning, causes, and fix sections. Your choice carries to other Fix Library guides on this device.

Practical explanations and page-specific steps. Recommended for most readers.

What the flag means

Crystal Ball flags robots.txt issues when crawler access may be blocked, unclear, or unavailable. Robots.txt gives cooperative crawlers path-level crawl rules; it is not an access-control system and does not reliably remove an already known URL from search.

Review the file when important pages or rendering assets are blocked by mistake. A missing robots.txt file is not automatically a defect if the site needs no crawl restrictions, although a valid file can provide explicit rules and a sitemap location.

Common causes

Common causes include staging rules left in place, old Disallow entries, platform or plugin output, security tools, blocked asset folders, or a file copied from another site. A scanner-specific block may also differ from the rules shown to major search crawlers.

What to check next

Check whether the issue appears on one page or every page using the same template. Review recent content, theme, plugin, deployment, and server changes, then compare the live output with a known-good page.

How to fix it

Confirm the warning first and make a backup before editing the live site. If the change involves templates, code, DNS, redirects, or server settings, switch to Intermediate or ask for help before proceeding.

  • Custom HTML/static: Create /robots.txt, avoid blocking important pages, and include the sitemap URL. Google's robots.txt introduction explains the syntax.
  • WordPress: WordPress may generate a virtual robots.txt. SEO plugins or server files can override it. Check that important content is not blocked.
  • Shopify: Shopify uses robots.txt.liquid for editable rules. Only change it when you understand the path being blocked.
  • Wix: Wix generates robots.txt automatically. Use page indexing controls instead of trying to hide pages with robots rules.
  • Webflow: Set robots.txt rules in Site settings. Do not block CSS, JS, or public pages that should be crawled.

AI search crawlers use separate controls

Do not treat every AI-related user agent as one bot. OpenAI documents OAI-SearchBot for ChatGPT search and GPTBot for potential model-training use as independent controls. Perplexity documents PerplexityBot for search visibility. Google says Googlebot and the usual Search eligibility and snippet controls apply to AI Overviews and AI Mode.

Robots.txt is only one layer. A CDN, WAF, JavaScript challenge, authentication wall, or 403 response can still block an allowed crawler. Use the dedicated AI Crawler Access and Citation Eligibility guide for crawler roles, verification, firewall checks, and the distinction between citation access and training controls.

Need help?

If the fix gets murky, visit Support and send the details. Bug reports and Crystal Ball questions are free support. Implementation and development work may be paid support, but we will tell you clearly before anything becomes paid. No surprise invoices from the tower.