AI Screening Calibration (QA)
A checklist with 6 steps: Define the pass-through rate for AI pre-screening before launch.
By InnovaAI ResearchPublished
What are the steps?
AI Screening Calibration (QA)
- 01
Define the pass-through rate for AI pre-screening before launch
Set a target percentage of candidates that should advance to human review, based on historical quality data from the client's past hires.
- 02
Run a shadow test on the last 50 closed roles
Feed past candidate resumes through the AI screener and compare its shortlist against the candidates who were actually hired, noting any mismatches.
- 03
Review AI screening decisions for demographic bias signals
Check whether the screener disproportionately filters out candidates from certain backgrounds, using the client's own diversity data as a baseline.
- 04
Interview the hiring manager on shortlist satisfaction
Ask whether the AI-selected candidates were a better fit than the previous manual shortlist, and document specific examples of misses.
- 05
Adjust screening thresholds based on the shadow test results
Lower or raise the AI's scoring cutoff until the pass-through rate aligns with the target, and re-run the shadow test to confirm the change.
- 06
Document the calibration settings and rationale for the client
Provide a one-page summary that lists the threshold, the bias check results, and the shadow test outcomes, so the client can audit the process.