Incrementality Test Design Review (QA)

Incrementality Test Design Review (QA)

1. 01

   Confirm the test has a single primary decision it will inform

   Write the decision in one sentence before touching configuration: for example, whether to shift 20% of paid social budget into retail media next quarter. Tests that answer several questions at once produce results no client will act on.

2. 02

   Verify the holdout or geo split is large enough to detect the effect the client cares about

   A 5% holdout on a channel contributing 8% of revenue rarely clears noise. Size the control group against the smallest lift worth acting on, not against what is convenient to carve out.

3. 03

   Check that control and treatment groups were assigned before campaign changes, not after

   Retrospective splits invite selection bias. If assignment happened mid-flight, flag the test as directional and say so in the readout rather than presenting it as causal.

4. 04

   Map every channel the test can influence, including halo effects on organic and direct

   A paid social holdout often depresses branded search and email revenue. Prescient AI and Measured both model halo explicitly; if your test does not, note the omission so the client does not read the gap as a win.

5. 05

   Reconcile the test window against the client's sales cycle and seasonality

   A 14-day test on a 45-day B2B pipeline measures lead flow, not revenue. Match the window to the metric you promised, or extend it and reset expectations in writing.

6. 06

   Document the attribution model the test will be judged against

   State whether the comparison uses last-click, multi-touch, or modeled contribution. Ruler Analytics and SegmentStream will report different numbers on the same spend, and the client will notice if the baseline shifts between readouts.

7. 07

   Set the stopping rule and the minimum detectable effect before launch

   Agree in advance on how many days or conversions trigger a call. Without a stopping rule, teams extend tests until the number looks favorable, which destroys the credibility of the whole measurement program.

8. 08

   Assign one named owner for the readout and a delivery date

   The readout is a client-facing deliverable, not an internal memo. Put the owner and date in the retainer scope so the analysis does not stall after the campaign ends.