Incrementality Test Design and Readout (QA)

Incrementality Test Design and Readout (QA)

1. 01

   Freeze the attribution baseline before any test cell goes live

   Export 90 days of channel-level spend, reported conversions, and CRM-closed revenue into a dated snapshot. Every later lift claim gets measured against this file, not against a dashboard that keeps recalculating.

2. 02

   Write the falsifiable hypothesis with a stated minimum detectable effect

   Example: paid social contributes at least 15% incremental revenue above what multi-touch attribution credits. If the test cannot detect a 15% swing at the client's current volume, say so before spending the budget.

3. 03

   Size the holdout from weekly conversion volume, not from a round number

   Pull four weeks of weekly conversions per geo or audience, then set holdout share so the expected control-group conversions clear the power threshold. A 10% holdout on a client doing 400 weekly conversions will read; the same share on 40 will not.

4. 04

   Confirm the measurement stack can survive the test window

   Check that server-side tracking, CRM sync, and any media mix model ingest are stable for the full flight. Platforms such as SegmentStream and Measured run incrementality alongside cross-channel attribution, but a broken CRM connector invalidates the readout regardless of the engine.

5. 05

   Run a pre-period equivalence check on control and treatment

   Compare the two cells on prior 30-day revenue, conversion rate, and seasonality index. If they diverge by more than a few points, re-randomize or move to a geo-matched design before launch.

6. 06

   Hold the test cell stable and log every mid-flight change

   Budget shifts, creative swaps, and bid strategy edits during the window contaminate lift. Keep a change log with dates so the readout can attribute anomalies to decisions rather than to the channel.

7. 07

   Reconcile the readout against the baseline and the model

   Present observed lift next to what last-click and multi-touch credited, plus any media mix estimate from a tool like Prescient AI. Where the three disagree, the gap is the finding, and it is what the client is paying to learn.

8. 08

   Publish a decision memo with the next test queued

   State the budget move the result justifies, the confidence band, and the follow-up test. Agencies that close the loop with a named next experiment keep the retainer conversation on evidence rather than on creative opinion.