7 November 2025 · Journal
Incrementality tests without a data science team
A win-back series that steals conversions from a receipt is not lift. You do not need a full experiment platform to stop celebrating cannibalisation.
Holdout Without a Data Team exists because most CRM operators we meet cannot staff an experimentation guild. They can, usually, exclude a random slice from a journey in their ESP. That is enough to start.
A holdout that survives Monday
Pick one sequence, not the whole catalogue. Freeze copy. Hold out a slice large enough that a quiet week still has events — we would rather a 12% holdout on one journey than a 2% sprinkle across everything. Measure the next action you actually care about (first basket, booking started, bill paid), not taps.
Write the influence definition before you look at results. If product is allowed to change the definition after seeing the chart, you do not have a test; you have a negotiation.
Ghost-control windows
When true randomisation is blocked — legal, vendor, or political — a ghost-control window compares the same audience in a matched clock period with the sequence paused. It is weaker. We still teach it because a documented weaker design beats an undocumented claim of “+18%”. Label the weakness in the report. Finance notices when you do not.
Significance theatre
Operators are often pressured to call every uplift significant. In class we refuse p-value cosplay without sample honesty. If the week is thin, say the week is thin. Extend the window or drop the claim. Module 5 of Signal to Send spends more time on that sentence than on any formula.
Cannibalisation is the failure mode we see most: a new sequence looks healthy because it intercepts users who would have converted from a transactional message. Your holdout should include those transactional sends in the baseline, or you will “win” by stealing from yourself.