Crystal Ball checks covered
CRAWL_MODE_SITEMAP, CRAWL_MODE_ORGANIC, page discovery options
What the flag means
This is less of an error and more of a choice. A sitemap crawl starts from URLs listed in your sitemap. An organic crawl starts from a page, usually the homepage, and follows internal links it can discover.
This matters because the two modes answer different questions. Sitemap crawl asks, “What does the site say should exist?†Organic crawl asks, “What can visitors and crawlers actually find?†Both maps can reveal different monsters.
Common causes
Differences appear when sitemaps include old URLs, pages are missing from navigation, links are JavaScript-only, redirects are listed in the sitemap, or important pages are published but not internally linked.
How to fix it
- Custom HTML/static: Use sitemap crawl when you have a clean sitemap of canonical URLs. Use organic crawl when you want to test discoverability through normal internal links.
- WordPress: Compare the SEO plugin sitemap against what the crawler finds through menus, body links, category pages, and footer links. Orphaned pages often show up here.
- Shopify: Use sitemap crawl for published products, collections, pages, and blogs. Use organic crawl to test whether important collections and pages are actually linked from the storefront.
- Wix: Use the sitemap for a known published-page list, then use organic crawl to check whether menus, buttons, and page links make those pages discoverable.
- Webflow: Use the Webflow sitemap for the intended public URL list. Use organic crawl to test navigation, CMS links, collection pages, and footer structure.
Need help?
If the fix gets murky, visit Support and send the details. Bug reports and Crystal Ball questions are free support. Implementation and development work may be paid support, but we will tell you clearly before anything becomes paid. No surprise invoices from the tower.