# Robots.txt: What It Does and How to Fix Problems > Learn how robots.txt controls crawler access, why it does not reliably prevent indexing, and how to avoid blocking important pages or assets. Canonical page: https://yourseowizard.com/education/fix-library/robots-txt/ Crystal Ball Fix Library Robots.txt is a crawler instruction scroll, not an invisibility cloak. By [Ian Garcia-Theriault](https://yourseowizard.com/about/ian-garcia-theriault/) Published June 29, 2026 - Updated July 30, 2026 ## Quick index - [Choose detail level](#reading-level) - [What the flag means](#flag) - [Common causes](#causes) - [How to fix it](#fixes) - [AI search crawlers](#ai-crawlers) - [Need help?](#support) - [Related fixes](#related) ## Crystal Ball checks covered `ROBOTS_BLOCKED`, `ROBOTS_UNKNOWN`, `robots.txt warnings` Guide detail ## Choose Your Technical Level This setting applies to the meaning, causes, and fix sections. Your choice carries to other Fix Library guides on this device. Basic Intermediate Advanced Practical explanations and page-specific steps. Recommended for most readers. ## What the flag means Crystal Ball flags robots.txt issues when crawler access may be blocked, unclear, or unavailable. Robots.txt gives cooperative crawlers path-level crawl rules; it is not an access-control system and does not reliably remove an already known URL from search. Review the file when important pages or rendering assets are blocked by mistake. A missing robots.txt file is not automatically a defect if the site needs no crawl restrictions, although a valid file can provide explicit rules and a sitemap location. ### Advanced diagnostic view Treat this finding as a diagnostic signal, not a verdict. Confirm it against the live URL and the exact Crystal Ball checks (ROBOTS_BLOCKED, ROBOTS_UNKNOWN, robots.txt warnings). Compare the final HTTP response, raw source, and rendered DOM where relevant, then determine whether the pattern is page-specific, template-wide, or introduced by infrastructure. ## Common causes Common causes include staging rules left in place, old Disallow entries, platform or plugin output, security tools, blocked asset folders, or a file copied from another site. A scanner-specific block may also differ from the rules shown to major search crawlers. ### What to check next Check whether the issue appears on one page or every page using the same template. Review recent content, theme, plugin, deployment, and server changes, then compare the live output with a known-good page. ### Advanced cause isolation Compare one affected URL with a known-good control at every layer that can own this signal: origin or CDN response, server rules, shared template output, client-side rendering, and third-party integrations. Record the differences before changing production; correlation narrows the investigation, but it does not prove the cause. ## How to fix it Confirm the warning first and make a backup before editing the live site. If the change involves templates, code, DNS, redirects, or server settings, switch to Intermediate or ask for help before proceeding. - **Custom HTML/static:** Create `/robots.txt`, avoid blocking important pages, and include the sitemap URL. Google's [robots.txt introduction](https://developers.google.com/search/docs/crawling-indexing/robots/intro) explains the syntax. - **WordPress:** WordPress may generate a virtual robots.txt. SEO plugins or server files can override it. Check that important content is not blocked. - **Shopify:** Shopify uses `robots.txt.liquid` for editable rules. Only change it when you understand the path being blocked. - **Wix:** Wix generates robots.txt automatically. Use page indexing controls instead of trying to hide pages with robots rules. - **Webflow:** Set robots.txt rules in Site settings. Do not block CSS, JS, or public pages that should be crawled. ### Advanced implementation workflow Capture a baseline and a rollback path, change the narrowest owning layer, and test representative URLs, templates, devices, and status variants. Validate the final response and rendered output, rerun the relevant checks, and monitor logs or search reports before applying the change broadly. ## AI search crawlers use separate controls Do not treat every AI-related user agent as one bot. OpenAI documents OAI-SearchBot for ChatGPT search and GPTBot for potential model-training use as independent controls. Perplexity documents PerplexityBot for search visibility. Google says Googlebot and the usual Search eligibility and snippet controls apply to AI Overviews and AI Mode. Robots.txt is only one layer. A CDN, WAF, JavaScript challenge, authentication wall, or 403 response can still block an allowed crawler. Use the dedicated [AI Crawler Access and Citation Eligibility guide](https://yourseowizard.com/education/fix-library/ai-crawler-access-citation-eligibility/) for crawler roles, verification, firewall checks, and the distinction between citation access and training controls. ## Need help? If the fix gets murky, visit [Support](https://yourseowizard.com/contact) and send the details. Bug reports and Crystal Ball questions are free support. Implementation and development work may be paid support, but we will tell you clearly before anything becomes paid. No surprise invoices from the tower. ## Related fixes - [AI crawler access and citation eligibility](https://yourseowizard.com/education/fix-library/ai-crawler-access-citation-eligibility/) - [Indexability problems](https://yourseowizard.com/education/fix-library/how-to-fix-a-page-that-cannot-be-indexed/) - [XML sitemaps](https://yourseowizard.com/education/fix-library/xml-sitemaps/) - [HTTP status and redirects](https://yourseowizard.com/education/fix-library/http-status-and-redirect-problems/) - [Schema and Indexability guide](https://yourseowizard.com/education/schema-indexability-seo/)