Crawl Budget Triage Before a Retainer Renewal (Retention)

Crawl Budget Triage Before a Retainer Renewal (Retention)

1. 01

   Pull 90 days of server log data and segment hits by bot identity

   Separate Googlebot, Bingbot, and AI crawlers such as GPTBot and ClaudeBot from human traffic so you can see which agents are actually spending crawl budget on the client's site.

2. 02

   Cross-reference logged URLs against the client's revenue pages

   Flag any product, pricing, or lead-capture URL that receives fewer than 5 bot hits per week; those gaps are the retention conversation, not a vanity metric.

3. 03

   Run a full crawl and reconcile the discovered URL set against the sitemap

   Tools like Sitebulb or Audisto surface orphaned and redirect-chained URLs that logs alone will not reveal, and the delta between crawl and sitemap is usually where index bloat hides.

4. 04

   Score each issue by revenue exposure, not by severity label

   A noindexed category page on a 40-page site matters less than a blocked facet on a 400,000-URL catalog; rank by estimated sessions at risk before writing the renewal deck.

5. 05

   Check whether AI answer surfaces cite the client for its top 10 branded queries

   Run those queries through Google AI Overviews, ChatGPT, and Perplexity, then record citation presence; AI Overviews now appear in 83% of branded searches, so an uncited brand is a visible gap to the client's competitors.

6. 06

   Package findings into a one-page exposure summary with a fix sequence

   List the three highest-exposure issues, the estimated engineering hours for each, and the order of operations; agencies that hand clients a sequenced plan renew more often than those that hand over a 60-page audit.

7. 07

   Schedule the remediation review 30 days out and assign an owner per issue

   Unassigned technical debt is the most common reason a retainer client churns; put a name and a date against every line item before the call ends.