Pre-Crawl Scope Agreement (Onboarding)

Pre-Crawl Scope Agreement (Onboarding)

1. 01

   Pull the client's current index footprint before promising any crawl volume

   Export indexed URL counts from Search Console and compare against the sitemap total. A 40,000-URL sitemap with 9,000 indexed pages changes the audit scope and the retainer line item.

2. 02

   Confirm which rendering mode the site actually serves to bots

   Fetch three representative templates with JavaScript disabled. If primary content only appears after client-side rendering, the crawl plan needs a rendering crawler rather than a raw HTML fetch.

3. 03

   Map the client's bot policy file and CDN rules against the crawlers you intend to run

   Check robots.txt directives, WAF rate limits, and staging password gates. A crawl that trips a rate limiter produces partial data that looks like a site problem.

4. 04

   Separate traditional search bots from AI crawlers in the access request

   GPTBot, ClaudeBot, and PerplexityBot are governed by different directives than Googlebot. Ask the client which AI crawlers they want allowed before you write recommendations that change their visibility posture.

5. 05

   Agree on the deliverable format and the issue-priority scale up front

   Decide whether the client wants a prioritized issue list, a client-ready PDF, or both, and define what counts as critical versus cosmetic. Sitebulb and Audisto both support client-facing report exports, so the format choice is a workflow decision, not a tool limitation.

6. 06

   Set the crawl window and notify the client's engineering contact

   Give a start time, expected duration, and the user agent string. Large-site crawls on JetOctopus or Audisto can run for hours and should not overlap a deploy.

7. 07

   Log the agreed scope, exclusions, and access credentials in the client record

   Record excluded subdomains, parameter rules, and who approved them. This becomes the baseline when the next audit cycle starts.