Failure PatternDecision layer
The Threshold Drift Trap: Why Fraud & Risk Signals Decay Quietly in Agency Retainers
Symptom: Block rates that looked healthy at onboarding creep upward by 1 to 3 points per quarter, and the client notices before the agency does. Root cause: Risk scores are calibrated against a traffic mix that changes constantly. A client running a seasonal promotion, entering a new country, or shifting paid spend to a different network will see the same score mean something different within weeks, and no one owns the recalibration.
By InnovaAI ResearchPublished
How do you recognize it?
- •Block rates that looked healthy at onboarding creep upward by 1 to 3 points per quarter, and the client notices before the agency does
- •Legitimate repeat buyers from a new region or device type start failing verification, generating support tickets the agency never attributes to the risk stack
- •The monthly fraud report still shows the same headline metrics as month one, so nobody re-tunes anything until a spike forces a review
- •Client-side developers have quietly raised a score cutoff in their own codebase, and the agency's documented configuration no longer matches production
- •Chargeback volume falls in the first 60 days, then flattens while false-positive complaints rise, and the retainer gets questioned at renewal
Why does it happen?
- •Risk scores are calibrated against a traffic mix that changes constantly. A client running a seasonal promotion, entering a new country, or shifting paid spend to a different network will see the same score mean something different within weeks, and no one owns the recalibration.
- •Agencies sell the integration as a one-time build and staff it as a project, not a monitored service. Once the SDK ships, the only person touching thresholds is whoever happens to see the alert, usually a client engineer with no fraud context.
- •Vendor signal quality is not uniform across geographies and device populations. A model tuned on one client's North American desktop traffic behaves differently on mobile-heavy or cross-border flows, and the degradation is gradual enough to escape a spot check.
- •Nobody instruments the cost of the false positives. Teams track blocked transactions and chargebacks but rarely track support contacts, manual review hours, or abandoned carts caused by a challenge, so the tradeoff stays invisible in the reporting.
How do you fix it?
- •Pull 90 days of decision logs and segment block and challenge rates by country, device class, and new-versus-returning visitor. Any segment with a block rate more than double the account average gets reviewed this week.
- •Stand up a weekly 30-minute threshold review as a named line item in the retainer, with one owner on the agency side and one on the client side. Put the decision log, not the vendor dashboard, in front of both.
- •Run a shadow period before changing any cutoff: score traffic at the current threshold and a proposed one in parallel for two weeks, then compare blocked volume against confirmed fraud and confirmed good customers.
- •Add a false-positive cost line to the monthly client report covering support tickets, manual review time, and abandoned checkouts, so the tuning conversation is about net margin rather than block counts.