AI Crawler Visibility Audit (QA)

AI Crawler Visibility Audit (QA)

1. 01

   Verify crawler access controls for AI bots

   Check robots.txt and meta tags for GPTBot, ClaudeBot, and PerplexityBot directives, confirming they are not blocked from key client pages.

2. 02

   Confirm raw HTML delivery of critical content

   Use a tool like Cite Hustle's audit or a curl command to ensure essential text is present in the initial HTML response, not injected by JavaScript.

3. 03

   Run a crawl with AI-bot tracking enabled

   Configure a crawler such as JetOctopus to log AI bot hits, then compare AI bot crawl frequency against Googlebot to spot coverage gaps.

4. 04

   Check indexation status in Google Search Console

   Pull the page indexing report and identify any client URLs that are crawled but not indexed, prioritizing those with organic traffic potential.

5. 05

   Test client visibility in major AI platforms

   Query ChatGPT, Perplexity, and Gemini with client-relevant questions to document whether the brand appears in AI-generated answers, noting any citation gaps.

6. 06

   Review server logs for bot access patterns

   Analyze log files for AI bot user agents, looking for anomalies like excessive crawl rates or blocked paths that could signal misconfiguration.

7. 07

   Document findings and prioritize fixes

   Compile a checklist of issues ranked by impact on AI visibility, from blocked crawlers to missing structured data, and assign owners for remediation.