Define the outcome before writing robots rules

Decide whether the site should appear in ordinary search, support generated search answers, permit snippets, allow browser-agent access, and permit potential training use. These outcomes may be governed by different crawlers, directives, and products.

  • Write a short policy for each intended use.
  • Map the policy to official crawler names and controls.
  • Keep legal or contractual requirements beside the technical configuration.
  • Review the policy when provider documentation changes.

Understand common controls

Robots.txt manages crawler access but does not remove a known URL from every search experience. Noindex is an indexing instruction that must usually be crawled to be seen. Snippet controls affect previews and supporting text. Provider-specific training crawlers can have distinct robots rules.

  • Do not use robots.txt as a substitute for access control on private data.
  • Allow crawling when a crawler must read a noindex directive.
  • Test whether server headers and HTML directives agree.
  • Use authentication for content that is genuinely private.

Use OpenAI's separate crawler identities carefully

OpenAI documents OAI-SearchBot for search discovery and GPTBot for potential training collection. A publisher can make different choices for these agents. Current documentation should remain the source of truth because names, products, and behavior can evolve.

  • Document the date each rule was reviewed.
  • Do not assume ChatGPT user browsing and server crawling are identical.
  • Check WAF behavior in addition to robots.txt.
  • Track referral traffic separately from crawler activity.

Publish a policy the team can maintain

A compact control matrix is easier to maintain than scattered rules copied from blog posts. Include crawler, provider, purpose, desired access, technical rule, verification method, owner, and last review date.

  • Store the matrix outside the public robots file.
  • Require review before blanket disallow or allow changes.
  • Test representative URLs after every change.
  • Remove obsolete crawler names only after confirming current guidance.

Official references

FAQ

Can I allow ChatGPT search but block GPTBot?

OpenAI documents separate controls for OAI-SearchBot and GPTBot, allowing publishers to make different search and training choices.

Does blocking a crawler make private information secure?

No. Robots.txt is public guidance for compliant crawlers, not an authentication or authorization system.

Can a noindex page still be linked by an AI product?

Provider behavior varies. OpenAI notes that a disallowed page known through another source may have its link and title surfaced in some circumstances; use current provider controls for the intended outcome.

Keep the Next Step Small

Use the related guides to confirm what the page needs. Ask for support only when the change reaches code, templates, or server settings you do not want to guess at.

Contact Your SEO Wizard