Mees Frima, partner success manager at Billy Grace, explains why incrementality tests are not to be feared – and what metrics you should really be looking at.
There’s a shoulder-clenching conversation that happens in most agency relationships at some point. Usually it starts with a client asking why revenue isn’t growing the way the ROAS figures suggest it should. The agency pulls the platform data. The numbers look fine, so heads are scratched. And then someone in the room says the quiet part out loud: maybe the channels aren’t delivering what they say they are.
“What would actually happen if we stopped advertising via this channel?” is the question most agencies know they should answer, but very few do. The answer just feels too risky to find out.
The fear of running the test is almost always bigger than what the test actually reveals. And the agencies that get past it tend not to look back.
How incrementality testing works in practice
There are several ways to measure incrementality. One of the most common is a geo-holdout test, and the methodology is more straightforward than it sounds. You take a channel you want to test. You pause it in one geographic region while running it normally everywhere else. You measure the difference in conversions, revenue, and traffic between the two groups over the test period. The gap between what you’d expect from the paused region and what you actually see is your incrementality: the genuine contribution of that channel to outcomes that wouldn’t have happened without it.
A geo-holdout test gives you something that channel-reported ROAS never can: a view of what your spend is actually causing, rather than what it’s happening alongside.
The signal looks different depending on the channel. For Google, the impact of pausing tends to show up directly in conversions. If search is genuinely incremental, you’ll see them fall in the holdout region. For Meta and other awareness channels, the signal is often in traffic rather than immediate conversions, because the intent Meta generates tends to convert later and elsewhere. That distinction matters for how you interpret results and what you do with them.
Case study: the workwear brand that was too cautious to pause
One of the clearest examples I’ve seen involved a global workwear brand with a significant organic presence on one of its key ad channels. The attribution data showed strong performance. The team was convinced that cutting the budget would cost them revenue, so the spend had become untouchable.
Our first question was straightforward: how much of what the channel was reporting was genuinely incremental — driven by the ads — and how much would have happened anyway through organic traffic? The brand had a strong enough organic presence that the answer wasn’t obvious.
We ran the test. The data showed that a meaningful portion of the conversions the channel was claiming were coming from users who would have found the brand organically regardless of whether the ads were running. The ads were accelerating some journeys, but they weren’t creating them. The result: the brand reallocated more than 10% of its total online advertising budget into channels where the incremental case was stronger.
The channel didn’t disappear from the plan. But it was right-sized. And the budget that moved out of it went to work harder elsewhere.
What happens when the test goes the other way
It’s worth being clear that incrementality testing doesn’t always tell you your spend is wasted. Sometimes it validates it.
We ran a test on Meta for a brand in a similar vertical. The conversion numbers stayed broadly stable during the holdout period. On the surface that might suggest Meta wasn’t incremental. But traffic to the client’s site fell. Sessions dropped. The audience Meta was reaching wasn’t converting immediately, but it was arriving — building familiarity, returning later through other channels, converting further down the funnel.
That’s an important distinction. A channel that doesn’t show up in short-term conversion incrementality can still be genuinely earning its budget through the demand it creates upstream. If you’d looked only at conversions, you’d have pulled the budget. The traffic data told a different story.
This is why running the test matters more than the result. You don’t run an incrementality test to prove a channel isn’t working. You run it to find out what it’s actually doing, so you can make a decision grounded in real-world evidence, on top of what your attribution data already tells you.
The harder part isn’t the methodology
In my experience, agencies don’t struggle to understand incrementality testing. Most practitioners get the logic quickly. What’s harder is what comes after: steering on the new data once you have it.
There’s a comfort in platform-reported numbers, even when you know they’re incomplete. They’re familiar. Clients have been looking at them for years. Changing the single source of truth in an agency-client relationship isn’t a measurement decision — it’s a trust decision. And trust gets built incrementally, which means the approach that works is to start small, prove the methodology in one area, and expand from there.
The agencies that do this well share a few things in common. They invest time in understanding the data before they take it to clients. They can explain not just what the test showed but why the methodology is sound. And they use the first test results not as a verdict but as the start of a conversation — one that, over time, repositions them from the team that manages the channels to the team that tells the client what’s actually working.
That’s the shift worth making. The test is just how you get there.
