An AI crawler policy should describe which services may access which public resources for which purpose. Start with that decision, then translate it into robots.txt and infrastructure rules. Copying a blanket “block AI” or “allow AI” list can produce a policy that does not match what your team intended.
This guide belongs to the technical AI readiness collection. It explains policy review; use the access audit to diagnose an actual failed request.
robots.txt is a crawler instruction, not authentication
robots.txt provides instructions to cooperating crawlers. It is not a password system, and publishing a disallow rule does not make the resource private. Google's robots.txt introduction explains this distinction and the limitations of using crawl restrictions to control search appearance.
Keep private material behind appropriate access controls. For public content, separately decide your search, training, and user-requested access preferences. A provider's published interpretation of those instructions is the relevant reference for that provider; do not assume identical behavior across products.
Understand the documented purposes
| Agent or token | Documented role to review | Policy implication |
|---|---|---|
| OAI-SearchBot | OpenAI search crawling | Decide search access independently from training preferences |
| GPTBot | Collection that may be used for foundation-model training | Review training use separately |
| ChatGPT-User | User-initiated page visits | Do not treat it as the automatic search crawler |
| PerplexityBot | Surfacing and linking pages in Perplexity search | Review public search access and provider verification |
| Perplexity-User | User-initiated page access | Review the provider's documented differences from crawling |
| Googlebot | Google Search crawling | Check the Search policy for the affected page |
| Google-Extended | A product control token for specified Gemini uses | Read its current scope rather than treating it as Googlebot |
These roles come from OpenAI's crawler reference, Perplexity's crawler reference, and Google's common crawler documentation. Google-Extended covers specified training and grounding uses; the documentation distinguishes it from Google Search inclusion. Recheck these references when updating policy because their scopes can change.
An illustrative policy choice
Suppose a hypothetical business wants its public pages available to OpenAI search but does not want GPTBot collection for training. The following demonstrates those two preferences only:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
This is not a complete production robots.txt. A real site may need private-path restrictions, separate subdomains, sitemap declarations, and other existing crawler groups. A broad allow in an agent-specific group can alter the intended restrictions if you assumed the wildcard group's rules would automatically carry over. Review the applicable rules for that agent and path before deployment.
Keep a copy of the current production file and annotate the desired outcome in your change request. Avoid combining a search policy change with unrelated path restrictions; a small change is easier to verify and reverse.
Review the complete request path
The relevant controls may be spread across your application, hosting platform, CDN, and firewall. An allow rule in robots.txt cannot bypass a login screen or a bot challenge. Equally, a successful browser visit does not tell you what a verified provider request receives.
Read the current provider guidance on verifying crawler traffic. A user-agent header alone is not proof of identity: anyone can send one. If a verified service is blocked, investigate the specific infrastructure rule and public paths involved rather than exempting every request with a matching string.
Do not hard-code an IP list copied from an old article into a permanent rule. Use the provider's current published verification information and establish an owner for maintenance.
Use a policy review worksheet
Copy this table and complete it before making a change. The entries are examples of decisions, not a recommendation for every business.
| Question | Example decision record |
|---|---|
| What content is affected? | Public product and documentation pages on the main domain |
| What is the desired use? | Search discovery allowed; one provider's training collection restricted |
| Which agent handles that use? | Record token and primary documentation URL |
| What rules apply now? | Save exact file contents, specific groups, and path matches |
| What infrastructure could intervene? | CDN bot rule, application login, and rate limit review |
| Who owns the change? | Website engineer with product and security policy input |
| How will it be verified? | Production response inspection and authorized request logs |
| When should it be reversed? | Unintended access, load, or restrictions on public pages |
Verify the policy, then reassess visibility
After deployment, fetch the production robots.txt, inspect its response, and check the intended agent/path combinations. Confirm the public page still serves useful content. Document any waiting period stated by the provider rather than expecting an immediate answer change.
Next, repeat your defined AI visibility audit. A successful policy deployment establishes that the rule changed; it does not establish that a future mention was caused by that change. Keep search access, training preferences, and observed answer visibility as separate reporting fields.
If the access policy is correct but brand information remains inconsistent, review structured data and entity consistency. If you need help connecting these findings to a monitoring workflow, bring the affected URLs to a BIU demo.