- Template item
Incrementality testing measures the sales a marketing channel actually caused, versus the sales that would have happened without it. You hold a slice of your audience out of the campaign, run the campaign to everyone else, and compare. The gap is the lift. Any revenue your attribution report credits to that channel but that the control group earned anyway was never incremental, it was going to happen regardless.
That gap is where budget decisions get real. Most teams move spend based on last-click or multi-touch reports that look like causation but are correlation. A clean lift test tells you whether the money changed the outcome. This guide walks you through scoping your first study: picking the right design, sizing a holdout you can actually afford, and writing the brief you hand to the analyst who runs it.
What incrementality testing actually measures
Incrementality testing measures causal lift: the share of conversions your campaign caused, not just the share it touched. You split your audience into a treatment group that sees the ads and a control group that doesn't. Both groups convert at some baseline rate. The difference between those rates, scaled to spend, is the incremental ROI.
Attribution reports can't do this. A last-click report tells you which touchpoint happened before the sale, not whether the sale needed the touchpoint. If a customer was going to buy anyway and clicked a retargeting ad on the way, retargeting gets 100% of the credit, and 0% of the lift. That's the trap.
The counterfactual is the whole point. What would have happened without this campaign? You can't observe that directly for the same person, so you approximate it with a control group that matches the treatment group on every observable dimension except exposure. Geo tests use geography as the split. Ghost-ad tests use the ad auction itself, the control sees a public service announcement in the slot your ad would have won. User-level holdouts split by user ID.
The output is a number with a confidence interval: your channel drove X% more conversions than it would have without the spend, give or take Y%. That's the number that should move budget. A positive attribution number with no lift means you're paying for conversions you'd get for free.
Attribution vs incrementality, the real difference
Attribution assigns credit for conversions that already happened. Incrementality tells you whether those conversions needed your spend. Both have a place, but they answer different questions, and most teams confuse them. Attribution models (last-click, MTA, MMM) all try to divide a known pie. A lift test asks whether the pie would be smaller without the campaign.
The three measurement approaches you'll hear about:
| Method | What it answers | Where it misleads |
|---|---|---|
| Last-click / MTA | Which touchpoints appeared on converter paths | Credits channels that touched the sale, not channels that caused it |
| Media Mix Modeling (MMM) | How spend correlates with revenue at a channel level over time | Needs 2+ years of data; smooths over short-term causality |
| Incrementality (lift) tests | Did this specific spend cause additional revenue, yes/no, how much | Costs you the holdout; narrow scope per test |
| Platform-reported ROAS | What the ad platform's model thinks happened | Marked its own homework; overcounts by 2-4× in most audits |
A healthy stack uses all three. MMM for strategic budget shape across quarters. Attribution for daily optimization. Lift tests for the big budget calls, before you double a channel or cut one. For a deeper look at the channel-mix fundamentals that sit upstream of this decision, start with SEO vs PPC.
Three study designs and when to use each
Three designs cover 90% of what you'll run: geo holdout tests, ghost-ad tests, and user-level holdouts. Pick based on the channel, the audience split you can control, and whether you need to test at the market level or the person level. All three produce a lift number with a confidence interval; they differ on what you have to give up to get it.
- Geo holdout test. Split your markets into treatment DMAs and control DMAs. Run the campaign in treatment markets, pause it in control. Compare conversion rates after 2-4 weeks. Best for broad-reach channels (TV, podcasts, CTV, upper-funnel paid social) where you can't track individuals cleanly. Needs enough markets to get statistical power, usually 20+ matched pairs.
- Ghost ads (placebo auction). The platform shows your ad to the treatment group when it would normally win the auction. For the control group, it tracks the same would-have-won moments but shows a PSA or nothing. Then it compares conversion rates between the two groups. Google Ads and Meta both offer this under names like "conversion lift" and "brand lift." See the Google Ads Help Center and Meta Business Help Center for the mechanics on each platform.
- User-level holdout. You randomly exclude a percentage of your targetable audience from a campaign and compare conversion rates between the exposed and held-out groups. Works best for owned-list channels: email, SMS, lifecycle push. You control the list, so you control the split cleanly.
Rule of thumb: use ghost ads when the platform supports it (lowest cost to run). Use geo holdouts when the platform doesn't, or when you need to measure a mix of channels in a market. Use user-level holdouts for anything you own. For a measurement perspective at the executive level, Think with Google and Harvard Business Review both publish ongoing work on causal measurement.
How to scope your first test (data, holdout, duration)
Scope your first test around a single channel you're already spending real money on, pick one KPI, and set the minimum detectable effect (MDE) before you launch. That order matters. Teams that pick the channel and the duration without pre-registering the MDE end up with underpowered tests that produce ambiguous results, and ambiguous results get rationalized toward whatever the team wanted to believe.
The pre-flight checklist:
- Pick the channel. The one with the biggest line item or the one you most suspect is inflated. Start with the budget decision you're actually facing.
- Pick the KPI. New-customer orders, qualified pipeline, revenue, trial starts. Not clicks. Not impressions. The thing finance cares about.
- Set the MDE. The smallest lift you'd care about. For a $200K/month channel, a 5% lift might be your break-even; a 2% lift is noise. Set the MDE at the business decision threshold, not the statistical floor.
- Pick the holdout percentage. 10% is the common starting point for user-level tests. 50/50 for geo tests (you need symmetric power on both sides). Smaller holdouts mean longer tests to reach significance.
- Pick the duration. Long enough to clear the longest conversion window for the KPI, plus a buffer. For a 14-day window, run at least 4 weeks. For B2B pipeline KPIs, 8-12 weeks.
- Pre-register the hypothesis and the stopping rule. Write down the MDE, duration, and decision rule before data arrives. Save it where you can't edit it. This is the only defense against post-hoc rationalization.
- Confirm clean data access. The analyst needs event-level data for both groups, with no leakage between them. If your event pipeline can't tag exposure, fix that before you launch, not after.
If you can't commit to leaving 10% of a market dark for 4 weeks, you can't run a clean test. That's not a flaw in the method; it's a signal that the budget call isn't ready. Buy an MMM report or run a smaller-stakes test first.
The hidden cost of a holdout (and how to size it)
A holdout costs you the incremental revenue you would have earned from the audience you kept dark. The simple math: holdout_cost = holdout_% × channel_spend × historical_ROAS × test_duration. For a $250K/month channel with a 3× historical ROAS, a 10% user-level holdout over 4 weeks costs roughly $75K in forgone revenue, if the channel is actually lifting at the rate the ROAS claims.
That "if" is the point. If the channel isn't lifting, the holdout cost is $0, because the group you held back was going to convert anyway. You find out which world you're in only by running the test.
A $75K holdout cost is cheap if it reveals that the channel is actually at a 1.2× lift instead of a 3× ROAS, because you'll reallocate ten times that amount in the next quarter. It's expensive only if you were already going to make the right call without the test. Most teams aren't.
Size the holdout against the budget decision, not the holdout cost. If the test's downside is "you lost $75K to learn the channel doesn't work," that's a bargain against the alternative of running it blind.
Reading the results without fooling yourself
A result is significant only against the pre-registered MDE and the pre-registered duration. If your test shows a 3% lift with a 95% confidence interval of -1% to +7%, you can't call it a win, the interval straddles zero. If it shows an 8% lift with a tight interval, you can. Pre-register these thresholds before data arrives. Peeking at results mid-test and stopping early inflates false positives.
Three things to watch for:
- False positives from early peeking. The temptation to call a test early when it's "trending up" is the single biggest cause of bad lift decisions. Set a stopping rule and honor it.
- False negatives from underpowered tests. If your MDE is 5% but your sample size can only reliably detect a 10% lift, a null result means nothing. The analyst should calculate power before launch, not interpret ambiguous results after.
- Channel contamination. If your treatment group also sees email and your control doesn't, you're not measuring one channel, you're measuring the stack. Hold other channels constant across groups.
A null result is a result. If a channel didn't lift, it didn't lift. Don't rerun until it does; that's p-hacking, and it will embarrass the team the first time an exec asks for the raw data. The honest report sounds like: "At the pre-registered MDE and duration, the test did not detect a lift. The upper bound of the confidence interval is X%, which is below the business decision threshold." That sentence saves careers.
When to hire fractional vs full-time for lift testing
Hire fractional when you'll run 2-6 tests per year and need the method set up correctly. Hire full-time when lift testing is a continuous program with monthly cadence across many channels. Most companies at the $5-50M revenue stage are in the first bucket. A vetted marketing analyst working 10-20 hours per week can scope and run quarterly tests without the overhead of a full headcount, which usually sits closer to $140K+ fully loaded.
The decision table:
| Situation | Hire type | Why |
|---|---|---|
| First lift test ever, 1-2 channels in scope | Fractional analyst (10-20 hrs/wk) | You need the method set up right; you don't need 40 hours of lift work per week |
| Quarterly budget review cycle with 3-6 tests/year | Fractional analyst (ongoing retainer) | Enough cadence to justify a steady relationship, not enough to feed a full-time role |
| Monthly lift program across 5+ channels, multi-brand portfolio | Full-time measurement lead + fractional support | At this volume the role becomes a function, not a project |
| You don't yet know which channel to test or whether you can afford the holdout | Fractional CMO or growth lead to triage first | Pick the strategic question before you staff the measurement |
If the owner authorizing the holdout isn't a measurement specialist, loop in a fractional CMO early. The CMO's job is to defend the holdout cost to finance when the first-week report looks noisy. That conversation goes better before the test launches than during it. For the hiring mechanics, see how to hire a marketing analyst and where measurement sits on the marketing org chart. The U.S. Bureau of Labor Statistics occupational data for market research analysts gives you a useful full-time salary floor for the comparison. If the test is a mix of paid-search and paid-social channels, the analyst should coordinate with the channel owners before pre-registration.
Specialist Brief, what to hand to your analyst
A good brief gives the analyst a channel, a KPI, an MDE, a timeline, and the data access they need. It also names a single business decision the test will inform. Without that decision, you're buying a report; with it, you're buying a budget move. Copy the block below into a doc, fill the brackets, and hand it to the analyst on day one.
Project: First incrementality test, [channel name]
Business question: Should the team [increase / cut / hold] spend on [channel] next quarter?
Decision threshold (MDE): [X]% lift (below this, move spend; above this, hold or scale)
Study design: [geo holdout / ghost ads / user-level holdout]
Holdout: [10%] of [audience / markets]
KPI: [new-customer orders / qualified pipeline / activated trials]
Duration: [4 / 8 / 12 weeks]
Data access needed:
- Event-level conversion data for exposed and control groups
- Spend data by day and by audience segment
- Ability to tag exposure flag in the data warehouse
Deliverables:
- Pre-registered test plan (signed off by [owner])
- Weekly diagnostic (sample size, early-warning data quality issues)
- Final report: lift %, confidence interval, recommended budget action, methodology appendix
Acceptance criteria:
- Pre-registration doc filed before launch
- Final report ships within 2 weeks of test close
- Methodology appendix is detailed enough that a second analyst could replicateShip that brief.
It's the deliverable that turns "you should test incrementality" into a budget-moving conversation with finance.
FAQ
Enough spend in the test channel to make a 3-5% lift statistically detectable in your conversion volume. In practice that often means a channel running at $50K+/month for consumer DTC or $150K+/month for B2B. Below that, confidence intervals get too wide to make a decision. Run an MMM-style report instead until spend is big enough.
Four weeks is the usual floor; eight weeks is safer for most B2B and considered-purchase categories. The test has to run long enough to clear the longest conversion window for the KPI and still leave two to three weeks of clean comparison data. Shorter tests risk mistaking day-of-week or promo-cycle noise for lift.
Not reliably, if the account doesn't generate enough conversions to detect your MDE at 80% power, the test is scientifically underpowered no matter how carefully you design it. The fix is to either batch multiple small accounts into one test, use a longer duration, or accept that the question is better answered by MMM at that scale.
They share the logic: random assignment, treatment and control, measured difference. The difference is scope. A/B tests compare two creatives or landing pages within a channel. Incrementality tests compare being exposed to the channel at all vs not. The holdout, not the variant, is the point.
Marketing scopes it, data science or an analyst runs it, finance signs off on the holdout cost. All three names belong on the pre-registration doc. If no one at the finance seat has signed off on leaving a slice of the market dark, the test will get killed mid-flight when the first-week numbers look weak. Get that signature on day zero.
Scope the test. Then hire.
The sequence matters: write the brief, pick the design, size the holdout, pre-register the MDE, then bring in a specialist to execute. Scoping yourself clarifies which hire you actually need and gives the analyst a day-one deliverable rather than a two-week discovery phase.
If you're ready to run your first lift test, MarketerHire can match you with a vetted marketing analyst in 48 hours. Hand them the brief above and start the pre-registration the same week.
