Question 1

How do you recognize it?

Accepted Answer

Monthly crawl reports show clean indexation while client pages never surface in ChatGPT, Perplexity, or Google AI Overviews answers for branded queries. Server logs reveal GPTBot and ClaudeBot hitting robots.txt disallow rules that nobody on the account has reviewed since onboarding. Audit deliverables list 200+ redirect chains and duplicate titles but contain zero findings on raw HTML delivery or JavaScript-gated content. Clients forward screenshots of competitors cited in AI answers and ask why their own pages are absent, and the agency has no baseline to answer with. Retainer scope documents crawl frequency and PDF report cadence but never names which crawlers, human or machine, are permitted to read the site.

Question 2

Why does it happen?

Accepted Answer

Audit templates were built for a Googlebot-only web, so crawler policy review stops at the sitemap and never extends to AI user agents. Deliverable economics reward volume of issues found over relevance, pushing teams toward 300-check crawls instead of a short list of access blockers. No one owns the robots.txt and rendering configuration between audits, so changes ship without a second review of what they expose or hide. AI visibility is measured, if at all, through referral traffic that undercounts influence occurring before the final click.

Question 3

How do you fix it?

Accepted Answer

Pull 30 days of server logs and list every bot user agent by hit count, then flag which ones are blocked or served degraded HTML. Run a five-query manual check per client in ChatGPT, Perplexity, and AI Overviews, screenshot the results, and attach the gap to the next report. Add a crawler access section to the standard audit template covering robots.txt directives, server-side rendering, and raw HTML parity. Quote AI crawler monitoring as a separate line item with its own hours so the work is funded rather than absorbed into the existing retainer.

More for Technical SEO