Crawl Budget and Log Analysis Review (QA)

Crawl Budget and Log Analysis Review (QA)

1. 01

   Pull server logs for the last 30 days and segment by bot type

   Separate Googlebot, Bingbot, and AI crawlers such as GPTBot and ClaudeBot to see which agents are actually hitting the client's site.

2. 02

   Compare crawl frequency against page importance and conversion data

   Flag pages that receive heavy crawl volume but drive little revenue, and identify money pages that are being under-crawled.

3. 03

   Check for crawl anomalies like sudden spikes or drops in bot traffic

   A spike may indicate a scraping attack or a misconfigured redirect loop, while a drop often signals a robots.txt change or server error.

4. 04

   Review robots.txt and meta robots directives for accidental blocks

   Ensure no critical pages are disallowed and that AI crawlers are not being blocked if AI visibility is a client goal.

5. 05

   Assess internal linking and site architecture to guide crawl distribution

   Use the log data to confirm that important pages are reachable within a few clicks and that orphan pages are being discovered.

6. 06

   Document findings and prioritize fixes by revenue impact

   Rank issues by how much they affect organic visibility and client revenue, and present the list in the next client report.