An AI crawler access audit asks a specific question: can the intended service obtain the useful public content at a particular URL? It does not ask whether the service will recommend the brand. Keep those questions separate so you can prove that a technical fix worked without claiming an unsupported visibility result.
Start with a known symptom: a public page returns an error, a cited description is stale, or a verified request is blocked. If all you have is a missing brand mention, access is one hypothesis among several. The broader technical readiness guide helps prioritize the investigation.
1. Define the target and preserve the baseline
Record the exact production URL, expected content, affected service, and observation time. Include the final hostname and whether the path normally redirects. Choose a few representative pages rather than scanning every URL without a hypothesis.
Save a copy of the original response and configuration before changing anything. A login page, challenge document, or branded error can return an apparently successful status. The response body matters as much as the code.
The following commands use a fictional domain. Replace it with a website you administer or are authorized to inspect.
curl -sS -D /tmp/example-page-headers.txt \
-o /tmp/example-page-body.html \
https://example.com/product
curl -sS -L -o /dev/null \
-w 'status=%{http_code} final=%{url_effective}\n' \
https://example.com/product
The first command saves headers and body for inspection. The second follows redirects and reports the final destination. Neither simulates a provider's actual network or proves crawler access.
2. Review the applicable access policy
Fetch the production robots.txt and identify the relevant group and path rules. Use the provider's current documented token, not a guessed browser user-agent name.
curl -sS -D /tmp/example-robots-headers.txt \
-o /tmp/example-robots.txt \
https://example.com/robots.txt
OpenAI publishes separate roles and verification information in its crawler reference. Perplexity provides its own crawler and user-fetcher documentation. Review those differences before calling a rule a search or training control.
Use the robots policy guide to check that an agent-specific rule does not undermine restrictions you thought were inherited from a wildcard group. Keep authentication and private-path protection separate from voluntary crawl instructions.
3. Compare policy with actual responses
Inspect the saved HTML for the important product description, factual details, and real links. Compare the same resource in a browser. If the useful content arrives only after client-side execution, note that as a delivery dependency to investigate.
For Google, its JavaScript SEO documentation explains the renderer's role. It does not specify how every other provider handles a page. Do not label a response “AI readable” solely because it works in a modern browser.
A request made with a copied crawler user-agent can test how your own server treats that header, but it cannot reproduce the provider's source IP, infrastructure, or collection behavior. Label such a result as a local diagnostic rather than a verified provider visit.
4. Investigate CDN and firewall evidence
If authorized logs show a failed provider request, preserve the response code, matched rule, requested path, time, and verification method. Check current provider-published verification data. User-agent text by itself is insufficient to identify a crawler.
Look for narrowly testable explanations: a regional access rule, a rate limit, an unintended login gate, or a challenge applied to the public documentation path. Adjust only the necessary policy after the site's infrastructure owner has reviewed the evidence.
Do not turn a diagnostic into broad bot protection removal. A scoped allow decision should have an owner, a documented purpose, and a way to reverse it if traffic or exposure changes unexpectedly.
5. Use a result ledger
This is a suggested audit template. The sample row is hypothetical.
| Field | Example entry |
|---|---|
| URL | Public integration documentation page |
| Service | A named search crawler, verified using its current documentation |
| Expected result | Public HTML containing the current integration description |
| Observed result | Challenge response matched to a specific CDN rule |
| Evidence | Authorized request log, saved response, and rule identifier |
| Proposed action | Exclude the verified service from that rule on the affected public path |
| Technical verification | Successful verified request with expected content |
| Remaining uncertainty | Whether or when the service will use the page in an answer |
Use “unknown” when logs do not establish a result. Absence of a logged request might reflect the observation window, logging coverage, or no visit; it is not proof of a block.
6. Verify before measuring the downstream outcome
After a change, repeat the original technical check and inspect the production configuration. Confirm redirects, content, and response behavior. Keep the saved evidence linked to the change so another engineer can verify the finding.
Then rerun the same AI visibility audit under the original monitoring conditions. A new citation is an observation; attributing it to the access change requires stronger evidence than timing alone. The content measurement guide describes ways to compare interventions with less ambiguity.
If access is healthy, move to content coverage or citation source analysis. For a BIU discussion, bring the affected URLs and saved evidence so the team can evaluate the actual problem instead of a generic readiness score.