Crystal Ball checks covered

ROBOTS_BLOCKED, ROBOTS_UNKNOWN, robots.txt warnings

What the flag means

Crystal Ball flags robots.txt issues when crawler access may be blocked, unclear, or unavailable. Robots.txt tells crawlers which paths they may request.

This is a problem when important pages, CSS, JavaScript, images, or sitemap paths are blocked by mistake. It can also confuse scans when the file is missing, malformed, or guarded by security rules.

Common causes

Common causes include staging protections left in place, old Disallow rules, Shopify robots.txt.liquid edits, WordPress plugin settings, security tools, blocked asset folders, or a file copied from another site.

How to fix it

  • Custom HTML/static: Create /robots.txt, avoid blocking important pages, and include the sitemap URL. Google’s robots.txt introduction explains the syntax.
  • WordPress: WordPress may generate a virtual robots.txt. SEO plugins or server files can override it. Check that important content is not blocked.
  • Shopify: Shopify uses robots.txt.liquid for editable rules. Only change it when you understand the path being blocked.
  • Wix: Wix generates robots.txt automatically. Use page indexing controls instead of trying to hide pages with robots rules.
  • Webflow: Set robots.txt rules in Site settings. Do not block CSS, JS, or public pages that should be crawled.

Need help?

If the fix gets murky, visit Support and send the details. Bug reports and Crystal Ball questions are free support. Implementation and development work may be paid support, but we will tell you clearly before anything becomes paid. No surprise invoices from the tower.