Experiments · 14 min read · Interactive

Is this channel actually driving sales, or would those sales have happened anyway? A geo-holdout read by difference in differences

I turned a channel up and sales went up. Then the finance lead asked whether the shop would have sold those anyway, and I did not know.

Andrei PopescuMedia buyer passionate about measuring outcomes, and the writer of this blog.July 2026
Written against the cited papers

It is the Thursday budget review, and I am the one at the screen. Two weeks ago I turned a channel up in half of a shop’s regions, and the chart behind me shows each of those regions selling about 15 more a day (the shop is synthetic, built so the right answer is known in advance). I am enjoying this. Then the finance lead, who has seen a lot of my charts, asks one quiet question.

“Would we have sold those anyway?”

I did not know. This post is about how to find out.

The short answer

The only way to know whether a channel drives sales, rather than riding along with sales that would have come anyway, is to switch it off somewhere and watch. The experiment is a geo-holdout, a design set out by Google researchers Vaver and Koehler. Split your regions at random, hold the change back from one group for a couple of weeks, and subtract that group’s change from the other’s. The season cancels, and what is left is what your channel caused.

When researchers at Facebook checked attribution-style estimates against experiments on the same campaigns, half were off by a factor of three. In my tests six regions caught a real effect only about 38 per cent of the time, forty regions about 95 per cent.

The method cannot give one certain number: even a perfect test returns a range, true only for the channel, regions and weeks you tested.

Watch · 9 min The whole method, start to finish: why attribution cannot settle cause, how a geo-holdout reads the lift as a range, and how big a test has to be.

Watch: the method in eight minutes

Prefer to read? Everything below covers the same ground in more depth, with the maths, a widget to play with, and the notebook and spreadsheet to run it yourself. (The narration is automated. Turn captions on with the CC button.)

The video draws on Gordon and colleagues (2019) in Marketing Science for the gap between attribution and experiments, Vaver and Koehler (2011) at Google for the design, Google’s Trimmed Match for pairing and the randomisation test, Eggers and colleagues (2024) for the rehearsal before launch, and Marketing Dive for the $2.41 incremental-return case. The full list is in the references below.

The question attribution cannot answer

Attribution models, from last click to the Markov removal effect, read the journeys you already have and tell you which channels sit on the paths that end in a sale. That is worth knowing. But sitting on the path is not causing the sale. A channel can appear on every winning journey and still be a passenger, riding along with people who would have bought through some other door anyway.

Researchers at Facebook ran large randomised experiments, then asked the usual attribution-style methods to estimate the same campaigns from the same data. Their verdict was blunt: “The observational methods often fail to produce the same effects as the randomized experiments” [1]. In half of their studies the estimate was off by a factor of three [1].

So attribution narrows the field to the two or three channels worth questioning. To settle one, you change it and watch.

The trap: everything moves at once

Why not just compare the treated regions before and after? Because in the same fortnight a hundred other things moved too. It got warmer. A competitor paused. Payday landed. A before-and-after reading hands all of that to your change.

The fix is almost embarrassingly simple. Pick some regions at random and hold them back. They live through the same weather, competitor and payday, everything except your change, so whatever they do is the would-have-happened-anyway number, seen directly instead of guessed. Your effect is whatever the treated regions did beyond that.

The coin flip is what makes it fair. Because chance alone decides which region goes where, the two groups end up alike (big regions and small, loyal markets and fickle ones), apart from the luck of the draw. Statisticians call them the treatment arm and the control arm. Google’s researchers set out this design for advertising, with whole regions as the thing you randomise and geo-targeted ads as the switch [2].

The formula, written out

Here is the maths. It takes two lines, and after them you can check every number in this post by hand.

The headline number is the treated regions’ change minus the held-back regions’ change, which is why it goes by the name difference in differences:

lift = ( mean(treatment, live) − mean(treatment, pre) ) − ( mean(control, live) − mean(control, pre) )

mean(treatment, …): the treated regions’ average daily conversions per region, before the change (pre) or while it ran (live).
mean(control, …): the same two averages for the held-back regions.
the two brackets: the first is the treated regions’ whole move, the change plus the season. The second is the season alone. Subtract, and the season cancels.

The second line turns that number into a range, using how much the regions disagree with each other:

SE = √( s²_T / n_T + s²_C / n_C )range = lift ± 1.645 × SE

s²_T, s²_C: how spread out the regions’ own changes are inside each group.
n_T, n_C: the region counts. The 1.645 makes it a 90% range. Fewer regions, or regions that react very differently to the season, give a wider range.

One more idea needs no formula. If your change did nothing, then which regions you called treated was arbitrary. So you shuffle the labels thousands of times, work out the lift each time, and see how often chance alone beats the lift you really saw. If almost never, the lift is real. This is a randomisation test, and Google’s paired-region design uses the same reasoning to size a test before it runs [3].

A worked example synthetic data

The notebook builds the shop from the opening scene. Into the fortnight after the change I bake two things: a season that lifts every region, and a smaller true effect that lifts only the treated ones. Can the method tell them apart?

The set-up	Value
Regions	20, split 10 treated and 10 held back
Before the change (the pre-period)	21 days
While it ran (the live window)	14 days
The season, felt by every region	about +8 conversions per region per day
The true effect, treated regions only	+6 per day

Daily conversions for the two arms: they overlap in the pre-period, both jump at launch, then the treatment arm sits above the control arm.

Before the change the two groups sit on top of each other. The treated group sits just 1.6 a day below, well inside the noise, which is the sign the coin flip balanced them.

The notebook also runs a dress rehearsal, which statisticians call a placebo test: the same sum on the weeks before launch, split at day 14, where nothing changed. It keeps every way the analysis could go wrong and removes only the effect [4]. A sound test finds nothing there, and this one does: about +1, with a range that covers zero. Only a test that passes its rehearsal deserves a live read.

Check before the live read	Result	What it should show
Gap between the groups before launch	−1.6 per day	close to zero
Rehearsal on the pre-period, split at day 14	+0.95, 90% range −2.05 to +3.94	a range that covers zero

After launch both groups drift up with the season, and the treated group steps above the other and stays there. That gap is the change.

Three bars: the naive read +15.1, the season +8.7, and the holdout lift +6.4 with its confidence range.

Now the live read. The naive before-and-after on the treated regions says about +15 a day, the number on my chart in the meeting. The held-back regions rose by about +9 with no change at all. That is the season, caught in the act.

Subtract one from the other and the holdout says +6.4 a day. The true effect I baked in was +6, so it found it. My chart had handed my change the entire season on top.

In a report I would write that as plus 6 a day, somewhere between +2 and +11. Never as a single number.

Is +6.4 real, or a lucky split? The notebook reshuffles which regions count as treated thousands of times.

A histogram of lifts from random re-splits, centred on zero, with the observed +6.4 lift far out in the right tail.

The real lift sits far out in the tail. A random reshuffle matches it, in either direction, only 2.6 per cent of the time, once in forty tries. The change did something.

Reading	Conv/region/day	What it is
Naive read (treated regions, before against after)	+15.1	the change plus the season
What the market did (held-back regions)	+8.7	the season alone
Geo-holdout lift (difference in differences)	+6.4, 90% range +2.2 to +10.7	the change alone
Reshuffles at least as extreme	2.6% of 5,000 (a two-sided p-value of 0.026)	how often chance alone gets there

Interactive Drag the true effect, the season and the number of regions. Watch the naive read chase the season while the holdout keeps only the change, and watch the range shrink as the test gets bigger. All the maths runs in your browser, and nothing leaves the page. It draws a fresh set of regions each time rather than replaying the worked example, so the same inputs give a number near that section’s rather than equal to it. That difference is the luck of the draw, which is the whole point.

Run one yourself

How many regions do you need?

The commonest way I see a geo test fail is that it was too small. Too few regions, and the range is so wide it covers zero, which reads as no effect but really means too small to tell. The notebook holds the true effect at +6 and grows the number of regions.

A power curve: with 6 regions the measured lift has a range of about plus or minus 6 that crosses zero, and by 40 to 60 regions it tightens to about plus or minus 2 around the true effect.

With six regions the range runs 6.5 either side of the answer, so it crosses zero. Across a thousand repeated runs a test that small catches a clear result 44 per cent of the time, a little under half. Its range is also too narrow at that size: the 1.645 is a large-sample multiplier, and with six regions the 90% range held the true effect 83 times in a hundred. From about two dozen regions it holds close to its 90.

By forty regions a clear read comes back 98 per cent of the time. By sixty it comes back essentially always.

So the honest first question about a geo test is not what the lift was. It is whether the test was big enough to find one. If your range covers zero, add regions or run longer before you conclude anything.

Regions	Range either side of the lift	Clear result
6	±6.5	44%
40	±2.7	98%
60	±2.2	essentially always

Caveats to watch out for

A geo-holdout is the cleanest read of cause in marketing, which is exactly why its failures are worth knowing. They hide behind a result that looks authoritative.

A wide range means too small, not no effect.

When the range covers zero, the test could not see an effect, usually because it had too few regions or ran too short. It is not evidence the change did nothing. Reporting an undersized test as proof that a channel does not work is how good channels get cut.

The answer does not travel.

A holdout measures this channel, in these regions, over these weeks, and nothing else. Last quarter’s read does not certify this quarter’s budget, and a lift on paid search says nothing about your email programme. A different channel or season needs a new test.

Spillover shrinks the lift.

The method assumes the change stays inside the treated regions. Two leaks break that. A held-back city next to a treated one catches some of the ads. And platform targeting can serve outside the regions you set: Meta’s Location Expansion is the sharp example, and it must be off on the test campaigns. A leak lifts the held-back regions too, so your true lift looks smaller than it is.

A few big, different regions can defeat the coin flip.

With only a handful of large cities that behave nothing alike, even a random split can land lopsided groups, and the simple difference is then biased. Serious implementations pair comparable regions, flip the coin inside each pair, and trim the most extreme pairs so a couple of giant cities cannot dominate [3]. When the groups cannot be balanced, the read falls back to building a stand-in for the treated regions out of the held-back ones, called a synthetic control. That is sturdier but no longer a pure experiment, so it is checked too: if the stand-in cannot match the treated regions before launch, the read is downgraded.

The held-back regions have to be left alone.

Holding the change off is not enough. If those regions get a compensating budget, a different bid strategy or any other attention, the comparison measures that too. Untouched means untouched for the whole window.

It costs something to run.

The held-back regions go without the change for the window. That is a real if temporary price, so a holdout is a decision made on purpose, for channels big enough to be worth settling.

Even a perfect holdout gives a range, not certainty.

The coin flip removes the season, the competitor and the payday, everything statisticians lump together as confounding. It does not remove the luck of the draw. A different random split of the same regions would have landed a little differently, and the range is that wobble. Anyone selling you a single certain lift number is selling you the wrong thing.

How the industry uses it

The biggest platforms and the serious independent firms all arrive at this method. Google formalised the modern version in 2011 [2] and built it into Google Ads as Conversion Lift based on geography, which measures the extra conversions a campaign drives [5]. Meta open-sourced its own version, GeoLift, built on synthetic controls [6].

Independent firms build businesses on it. Recast sells a geo product that separates the channels driving growth from the ones claiming credit they did not earn [7], and treats a clean test as the yardstick for everything else: “well-run incrementality experiments are the best estimate that a marketer has for true incrementality” [8]. Haus describes the same random split of markets into advertised and held back [9], and Measured sells geo-matched split tests [10].

The results are concrete. One public retail-media case study reported a $2.41 incremental return on ad spend from a matched-market test with over 5.5 million impressions [11]. In plain words, about $2.41 of extra sales for every dollar spent.

This is the method the channel-attribution post pointed you toward. Attribution finds the channels worth questioning, and a holdout settles whether one of them is really driving sales.

Run it on your own data

The downloads run the whole method on your own computer, and your data never leaves it. They take a table of daily conversions per region:

column	meaning	example
`geo`	region name or code	`US-CA`
`date`	the day, ISO format	`2026-05-01`
`arm`	`treatment` or `control`	`treatment`
`conversions`	that region’s conversions that day	`47`

You also give them the launch date, which splits every region’s history into before and during. They check the columns, run the same sum and, the part that matters, warn you in plain words when the data is too thin to trust. They flag fewer than five regions per group, a live window under seven days, groups that look unbalanced before the change, and the big one: a range that covers zero, which they label too small to tell, never proven to do nothing.

The spreadsheet starts one step later than the notebook. It takes one number per region, that region’s change (its average day during the test minus its average day before), so build those with a pivot table first or let the notebook read the daily file.

Notebook · PythonThe full method, executableRuns the difference in differences, the randomisation test and a power curve, regenerates every figure in this post, and reads your own geo test with plain-words warnings when it is too thin. It also carries a raw-log consolidator and a GA4 and BigQuery recipe.Spreadsheet · ExcelThe same maths in live cellsA playable workbook: the worked example, plus a your-data sheet where you paste your region numbers and the lift, the range and a sufficiency flag recompute in the cells. Every cell is a formula, so nudge one and the result moves.PDF · for your AIGive it to your assistantA brief to hand to ChatGPT or Claude. It walks you through pulling your data, adding the arm column and running the notebook, one step at a time, with no code experience needed.

The outcome has to come from a source that sees every region. If the held-back regions have the ads switched off, the ad platform reports nothing for them. Take the outcome from store sales, CRM orders or site analytics by region.

Getting your data into that shape

Most analytics exports are a long log of single conversions, each stamped with a region and a date. The notebook’s consolidate_conversions helper rolls that log up into daily counts per region and fills in the days with no conversions (a day with no sales is real information, not a missing row). Then you add the arm column from the split you actually ran.

If your site sends conversions to Google Analytics 4 with the free BigQuery export on, the notebook ships a ready query that pulls one row per conversion, with region and date. Swap in your conversion event and region field, run it, and hand the output to the consolidator. My platform does this assembling automatically, and the notebook shows you how to do it by hand.

This part is technical, but you do not have to do it alone. Hand the give-it-to-your-AI brief to ChatGPT, Claude or any capable assistant, and it will walk you through pulling your data and running the test one step at a time, even if you have never written a line of code.

Common questions

Is a geo-holdout the same as an A/B test?

It is the same idea, moved up a level. An A/B test randomises people. A geo-holdout randomises regions, because for a whole channel’s budget you usually cannot split individual people, but you can split cities or states. When you can randomise people cleanly (a platform lift test, an experiment on your own site), that is even better.

The range on my result includes zero. Did my change fail?

Usually not. Most often the test was too small to tell. Check the number of regions and the length of the window, then add regions or run longer and read it again. A range that covers zero says the test could not measure the effect, never that the change does nothing.

Do the control regions lose money?

A little, and on purpose. You hold the change back from them for the window, which is why you run a holdout only for channels big enough to be worth settling.

How many regions do I need?

Enough that the range clears zero for the effect you expect. A handful leaves a range too wide to conclude anything, and a few dozen usually gives a clean read. With a small country’s worth of regions pairing helps, but the synthetic-control fallback is weakest exactly there. With fewer than nine held-back regions its own rehearsal test cannot call a lift real at the 90% level, and a poor match before launch drops the read a grade. On real revenue, a test over a demonstration store’s sixteen biggest countries, measured for my platform, could only have detected very large effects. Size the test before you run it.

Can I reuse a holdout result for another channel or another quarter?

No. The reason a holdout is trustworthy (it measured this thing directly) is also why it does not transfer. Reusing last quarter’s paid social lift to justify this quarter’s search budget is exactly the mistake the method exists to prevent.

Is this better than my attribution model, then? Should I drop attribution?

They do different jobs. Attribution is cheap and runs on data you already have, so it narrows the field to the two or three channels whose credit looks surprising. A holdout is slow and costs something, so you spend it on those few. Use attribution to choose what to test, and the holdout to settle it.

What if my channel has no regions to split, like a CRM or email tool?

Randomise contacts instead. Split your list at random, send the change to one half, and read the same difference. Watch for spillover between people (a treated customer telling a held-back one), and put whole households or accounts on the same side where you can.

Try it on your own test

Pick the one channel your attribution report credits most and you trust least. Split your regions at random, hold the change back from half of them for two weeks, and pull daily conversions by region from a source that sees them all. Run the notebook and read the range before the lift. Next time a finance lead asks whether you would have sold those anyway, you will have the held-back regions to point at.

Running this on your own account

The worked example above is synthetic. The notebook runs on your data instead, and every number it gives you carries its range. If you want a hand with the method, or with the paid media underneath it, I take on freelance work, and I answer questions about anything published here.

References

[1] Gordon, B. R., Zettelmeyer, F., Bhargava, N. and Chapsky, D. (2019). A comparison of approaches to advertising measurement: Evidence from big field experiments at Facebook. Marketing Science, 38(2), 193 to 225 (accessed 19 July 2026). https://doi.org/10.1287/mksc.2018.1135
[2] Vaver, J. and Koehler, J. (2011). Measuring Ad Effectiveness Using Geo Experiments. Google Inc. (accessed 19 July 2026). https://services.google.com/fh/files/blogs/geo_experiments_final_version.pdf
[3] Chen, A., Longfils, M. and Remy, N. (2021). Trimmed Match Design for Randomized Paired Geo Experiments. Google, open-source library trimmed_match (accessed 19 July 2026). https://github.com/google/trimmed_match
[4] Eggers, A. C., Tuñón, G. and Dafoe, A. (2024). Placebo Tests for Causal Inference. American Journal of Political Science, 68(3), 1106 to 1121 (accessed 19 July 2026). https://doi.org/10.1111/ajps.12818
[5] Google. Set up Conversion Lift based on geography. Google Ads Help (accessed 19 July 2026). https://support.google.com/google-ads/answer/14097193
[6] Meta Open Source. GeoLift: measuring incrementality at a geo level. GitHub (accessed 19 July 2026). https://github.com/facebookincubator/GeoLift
[7] Recast. GeoLift by Recast (accessed 19 July 2026). https://getrecast.com/geolift-by-recast/
[8] Recast. How Recast incorporates experiments. Recast documentation (accessed 19 July 2026). https://docs.getrecast.com/docs/experiments
[9] Haus. Geo experiments: the fundamentals. Haus blog (accessed 19 July 2026). https://www.haus.io/blog/geo-experiments-the-fundamentals
[10] Measured. Incrementality Testing (accessed 19 July 2026). https://www.measured.com/incrementality-testing/
[11] Marketing Dive (2026). Mondelēz lifts sales as Albertsons tackles in-store retail media measurement (accessed 19 July 2026). https://www.marketingdive.com/news/mondelez-lifts-sales-albertsons-tackles-in-store-retail-media-measurement/808842/

From Stochastic Strata. I build marketing measurement that shows its working: every number carries its range, and when the data cannot answer a question, I say so and tell you what would. The worked example here is synthetic. The method is the real thing, and you can run it on your own data with the notebook or spreadsheet above. Nothing in this post is financial advice.