Experiments · 13 min read · Interactive

Is this channel actually driving sales — or would those sales have happened anyway?

You put more budget into a channel and sales went up. The dashboard is happy. But did the spend cause those sales — or would they have arrived anyway?

The question attribution cannot answer

Attribution models — last-click, or the Markov removal effect — are useful. They read your existing journeys and tell you which channels are structurally important: which ones sit on the paths that convert. That is worth knowing. But structural importance in data you already have is not the same as causing extra sales. A channel can appear on every winning journey and still be a passenger — riding along with demand that would have converted through some other path anyway.

This is not a hypothetical gap. When researchers at Facebook compared observational attribution against randomised experiments on the same advertiser's data, the two often disagreed: “The observational methods often fail to produce the same effects as the randomized experiments, even after conditioning on extensive demographic and behavioral variables” [GORDON-2019]. The disagreement was not small — in half of their studies, the estimated lift was off “by a factor of three across all methods”[GORDON-2019]. Attribution narrows the field to the two or three channels worth interrogating. To actually settle one, you change it and watch.

The trap: everything moves at once

Here is why you cannot just look at the treated regions before and after. Suppose you turn a channel up in a set of regions and conversions rise. In that same window, a hundred other things also moved: it got warmer, a competitor paused, payday landed, the category had a good fortnight. A simple before/after reading of the treated regions hands all of that to your change. It cannot tell the season apart from the spend.

The control arm is the fix, and it is almost embarrassingly simple. Pick regions at random to hold back. They feel the same season, the same competitor, the same payday — everything except your change. So whatever they do over the window is the “would have happened anyway” number, observed directly. The lift is what the treated regions did beyond that.

Randomising which regions go in which arm is what makes this work. Because the split is a coin flip, the two arms are alike in every way — big regions and small, loyal markets and fickle ones — except by chance. Google's foundational write-up puts the design plainly: “non-overlapping geographic regions are randomly assigned to a control or treatment condition, and each region realizes its assigned condition through the use of geo-targeted advertising”[VAVER-KOEHLER-2011].

The formula, unpacked

The headline number is a difference in differences: each arm's own before/after change, then the control arm's change subtracted from the treatment arm's.

Two more pieces make it a measurement rather than a single point. The range comes from how much the regions disagree with each other:

And the significance comes from the randomisation itself. If the change truly did nothing, then which regions we called “treatment” was arbitrary — so we re-shuffle the labels thousands of times, recompute the lift each time, and ask how often a random re-split beats the lift we actually saw. If almost never, the lift is real. This design-based test is the same logic the trimmed-match geo-design library uses; its procedure “is equivalent to the standard permutation test” [CHEN-LONGFILS-REMY-2021].

A worked example synthetic data

The notebook builds a small advertiser's test: 20 regions, split 10 treatment and 10 control, with a 21-day pre-period and a 14-day live window. All of it is synthetic and labelled as such — the point is the method, not the numbers. Into the live window we bake two things at once: a market-wide seasonal lift of about +8 conversions per region per day that hits both arms, and a true tested-change effect of +6 per day that hits the treatment arm only. That is the trap, built on purpose. Can the method tell them apart?

In the pre-period the two arms sit on top of each other — that overlap is the evidence the randomisation worked (the pre-period gap is just −1.6, well inside the noise). It doubles as a placebo check: run the same contrast over the pre-period, where the change has not happened yet, and only trust the live read if the arms were indistinguishable beforehand [EGGERS-2024]. At launch both arms jump: that shared jump is the season. The treatment arm then keeps climbing above the control arm, and that gap is the change.

The true effect we baked in was +6, so the holdout recovered it; the naive read over-credited the change by +8.7 — the entire season. Is +6.4 real, or a lucky split? The randomisation test re-shuffles which regions were “treatment” 5,000 times and asks how often a random re-split would produce a lift this big by chance:

The observed lift sits out in the tail of that null distribution — a random re-split beats it only about 2.6% of the time (a p-value of 0.026). The change did something.

How many regions do you need?

The single most common way a geo test “fails” is that it was too small: too few regions, and the range is so wide it covers zero — which reads as “no effect” but really means “too small to tell.” The notebook holds the true effect fixed at +6 and grows the number of regions:

With 6 regions the lift lands around +4.6 but the range runs roughly ±6.4 — it crosses zero, and across repeated runs a test this small catches a clear result only about 38% of the time. By 40 regions the range is down near ±2.6 and a clear read comes back about 95% of the time; by 60 it is about ±2.2 and essentially always. This is why the honest first question about a geo test is not “what was the lift” but “was the test big enough to have found one.” If your range covers zero, add regions or run longer before you conclude anything.

How the industry uses it

The geo experiment is not fringe — it is the measurement the largest platforms and the serious independent shops all converge on. Google introduced the modern form (“One approach that Google has successfully employed to measure advertising effectiveness is geo experiments”[VAVER-KOEHLER-2011]) and productised it inside Google Ads, where Conversion Lift “based on geography” lets advertisers “measure the incremental conversions driven by your Google Ads campaigns” [GOOGLE-CONVERSION-LIFT]. Meta open-sourced its own version, GeoLift, “an end-to-end geo-experimental methodology based on Synthetic Control Methods used to measure the true incremental effect (Lift) of ad campaign”[GEOLIFT-META].

Independent measurement firms build their businesses on it. Recast sells a geo product that “measures the incremental conversions driven by marketing activity — separating true growth drivers from channels claiming credit for conversions they didn't actually drive”[RECAST-GEOLIFT], and treats a clean test as the yardstick everything else is calibrated against: “we believe that well-run incrementality experiments are the best estimate that a marketer has for true incrementality”[RECAST-EXPERIMENTS]. Haus describes the identical design — “markets across a specific country are randomly assigned to receive either the treatment (advertising on a specific channel) or the control (no advertising)”[HAUS-GEO] — and Measured runs “geo-matched and first-party split tests” [MEASURED-INCR]. The reads are concrete: one public retail-media case study reported “over 5.5 million impressions and a $2.41 incremental return on ad spend” from a matched-market test [MONDELEZ-ALBERTSONS].

This is the method the channel-attribution post pointed you toward: attribution finds the channels worth interrogating; a holdout is how you settle whether one of them is really driving sales.

Run it on your own data

The downloads run entirely on your machine — your data never leaves it. They take a table of daily conversions per region:

You also tell them the launch date — the first day the change went live — which splits every region's series into pre-period and live window. They validate the columns, run the same difference-in-differences, and warn you in plain words when the data is too thin to trust instead of printing a confident number. As rough rules of thumb they flag fewer than five regions per arm, a live window under seven days, arms that look unbalanced before the change, and the big one — a range that covers zero, which they label “too small to tell,” never “proven to do nothing.”

Getting your data into that shape

Most analytics exports are not a tidy region-by-day table — they are a long log of individual conversions, each stamped with a region and a date. The notebook includes a consolidate_conversions helper that does the tedious middle step: it rolls that raw log up into daily counts per region and fills in the zero-conversion days (a day with no sales is real signal, not a missing row). After that you add the arm column from the split you actually ran and set the launch date.

Where does the raw log come from? If your site sends conversions to Google Analytics 4 with the free BigQuery export on, the notebook ships a ready query that pulls one row per conversion, stamped with region and date. Swap in your conversion event and your region dimension, run it, hand the output to the consolidator, and you have the table above. Assembling and stitching that data is the part the platform automates; here we show how to do it by hand.

This part is technical — a database query and a notebook — but you do not have to do it alone. Download the give-it-to-your-AI brief and hand it to ChatGPT, Claude, or any capable assistant. It carries everything the assistant needs to walk you through getting your data and running the analysis, one step at a time, even if you have never written a line of code.

Common questions

From Stochastic Strata. We build marketing measurement that shows its working: every number carries its range, and when the data cannot answer a question, we say so — and tell you what would. The worked example here is synthetic; the method is the real thing, and you can run it on your own data with the notebook or spreadsheet above. Nothing in this post is financial advice.