Collect the minimum useful fields
A practical crawler log includes time, method, path, status, response bytes, user agent, edge or request ID, and a privacy-safe client classification. Query strings and IP addresses may contain sensitive information and should have explicit retention and access rules.
- Document why each retained field is needed.
- Redact secrets and unnecessary query parameters.
- Restrict raw log access.
- Set deletion schedules before collection begins.
Classify identity with confidence levels
User-agent strings are claims. Label traffic as verified when official methods confirm it, claimed when only the string matches, and unknown otherwise. This prevents a dashboard from converting spoofed traffic into impressive crawler demand.
- Use official IP or DNS verification where available.
- Cache verification results for an appropriate period.
- Record the verification method and timestamp.
- Do not publish raw addresses in reports.
Report behavior that can drive action
Useful views include status distribution, blocked important paths, repeated redirects, crawl concentration, asset requests, response size, and changes after deployment. Total requests alone rarely explain whether access is healthy.
- Separate search, training, agent, and unknown traffic.
- Alert on 401, 403, 404, 429, and repeated 5xx responses.
- Compare canonical and noncanonical route requests.
- Review whether a crawler receives the intended page body.
Keep crawler and referral reports separate
Crawler requests are machine access. Referral sessions are human visits following links from an AI product. They can be related, but combining them produces a misleading traffic report.
- Use analytics and request context for referral classification.
- Track known UTM parameters such as ChatGPT referrals.
- Keep crawler hits out of human engagement metrics.
- Publish aggregate trends only after privacy review.
Official references
- Google: Verify Googlebot and other Google crawlers: Google documents reverse and forward DNS verification rather than trusting a user-agent string alone.
- OpenAI: Publishers and developers FAQ: OpenAI explains OAI-SearchBot access, ChatGPT referral tracking, model-training controls, and agent accessibility.
FAQ
Can I identify an official bot by user agent alone?
No. User agents can be spoofed. Use the verification method published by the provider when available.
Are crawler requests website traffic?
They are server traffic, not human sessions. Keep them separate from visits, engagement, and conversions.
How long should crawler logs be retained?
Retain them only as long as needed for security, operations, and approved research, subject to applicable privacy obligations and published policies.
Keep the Next Step Small
Use the related guides to confirm what the page needs. Ask for support only when the change reaches code, templates, or server settings you do not want to guess at.
Contact Your SEO Wizard