Shortlist Quality Review (QA)

Shortlist Quality Review (QA)

1. 01

   Pull the last 20 auto-ranked shortlists and compare them against the hiring manager's final interview decisions

   Track how many ranked candidates reached a live interview and how many hires came from outside the top 5. A shortlist that misses hires is a rubric problem, not a volume problem.

2. 02

   Re-read the scoring rubric against the actual job description for each active client role

   Rubrics drift when a client quietly changes requirements mid-search. Confirm the criteria in the tool match the criteria the client is now screening for.

3. 03

   Sample five rejected candidates per role and read their full responses before accepting the rejection

   Automated scoring penalizes accents, non-linear career paths, and non-native phrasing. A human read of five rejections per role catches the false negatives cheaply.

4. 04

   Insert a human review checkpoint before any ranked shortlist reaches a client-facing portal

   Fewer than half of agent-generated insights lead to meaningful action at surveyed organizations, so treat machine rankings as a draft that a recruiter signs off on, not a deliverable.

5. 05

   Check that every candidate received the same question set and the same response window

   Uneven question sets or a 24-hour window for one applicant and a 72-hour window for another invalidate comparison across the pool.

6. 06

   Log the pass-through rate, median screen-to-shortlist time, and human override rate per client

   These three numbers are the evidence base for renewing a fractional recruiting retainer. Override rate above roughly 30 percent signals the automation is not yet trustworthy for that role type.

7. 07

   Flag senior and niche roles for live recruiter calls and remove them from the async queue

   Candidates for director-level and specialist positions expect a human on the first touch. Route them to a live call and keep async screening for volume and junior roles.