{ "cells": [ { "cell_type": "markdown", "id": "3e45331d", "metadata": {}, "source": [ "# Did my change actually work?\n", "\n", "**Interrupted time series, end to end.** This notebook builds a small advertising\n", "account, makes one change to it, and then measures that change two ways: the\n", "before/after average almost everyone reaches for, and a counterfactual that asks\n", "what the metric *would have done* if nothing had been changed.\n", "\n", "The two answers disagree, and the before/after one is wrong. That disagreement is\n", "the whole point.\n", "\n", "**The data here is synthetic and labelled as such.** It is generated below under\n", "one fixed seed, so every figure and every number in the accompanying article\n", "regenerates from this notebook exactly. Part two of the notebook runs the same\n", "analysis on **your** data - it never leaves your machine." ] }, { "cell_type": "code", "execution_count": 1, "id": "58537dde", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:42:52.958924Z", "iopub.status.busy": "2026-10-24T15:42:52.958597Z", "iopub.status.idle": "2026-10-24T15:43:06.266868Z", "shell.execute_reply": "2026-10-24T15:43:06.262377Z" } }, "outputs": [], "source": [ "import json\n", "from pathlib import Path\n", "\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import pandas as pd\n", "from statsmodels.stats.diagnostic import acorr_ljungbox\n", "from statsmodels.tsa.statespace.sarimax import SARIMAX\n", "from statsmodels.tsa.stattools import adfuller\n", "\n", "for d in (\"figures\", \"data\"):\n", " Path(d).mkdir(exist_ok=True)\n", "\n", "SWEEP_DRAWS = 10_000 # the per-window sweep: same draw count as the headline interval\n", "SEED = 20260903 # one seed; every number below regenerates from it\n", "N_PRE, N_POST = 56, 21 # days before and after the change\n", "rng = np.random.default_rng(SEED)\n", "\n", "PAPER, INK, MUTED, LINE = \"#fdfcf9\", \"#1f2124\", \"#565c66\", \"#e6e2da\"\n", "GREEN, BLUE, WARM = \"#0a8f5f\", \"#2a78d6\", \"#b06a2c\"\n", "FIGDIR = \"figures\"\n", "TOLERANCE = 1e-4\n" ] }, { "cell_type": "markdown", "id": "3d10c6b9", "metadata": {}, "source": [ "## 1. An account with a problem\n", "\n", "A lead-gen account. Cost per lead has been drifting up for weeks - auction\n", "pressure, nothing dramatic. The team cuts the target CPA to try to pull it back.\n", "\n", "Two things matter about how this series is built. The level **wanders** rather\n", "than sitting on a fixed line, which is how an auction-priced metric actually\n", "behaves. And the climb was already happening *before* anyone touched anything -\n", "which is exactly the trap the before/after average walks into." ] }, { "cell_type": "code", "execution_count": 2, "id": "a10311a2", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:43:06.289167Z", "iopub.status.busy": "2026-10-24T15:43:06.288564Z", "iopub.status.idle": "2026-10-24T15:43:06.338019Z", "shell.execute_reply": "2026-10-24T15:43:06.337268Z" } }, "outputs": [], "source": [ "BASE_CPA = 31.50 # EUR where the pre-period opens\n", "DRIFT = 0.17 # EUR/day the climb the account was already on\n", "DOW_AMP = 2.20 # EUR weekday/weekend swing\n", "STEP_SD = 1.05 # EUR day-to-day random-walk step\n", "TRUE_EFFECT = -5.60 # EUR the change actually delivered per day\n", "\n", "def _dow(idx):\n", " \"\"\"Weekly shape: cheaper mid-week, dearer at the weekend (lead-gen).\"\"\"\n", " return DOW_AMP * np.sin(2 * np.pi * (idx % 7) / 7.0 + 0.9)\n", "\n", "def generate_series(seed=SEED, n_pre=N_PRE, n_post=N_POST, true_effect=TRUE_EFFECT):\n", " \"\"\"Daily CPA as a random walk with drift plus a weekly shape, and a real\n", " post-intervention level shift.\n", "\n", " The level WANDERS rather than reverting to a fixed line - which is how an\n", " auction-priced metric actually behaves, and why the model differences the\n", " series before fitting. Generating a deterministic trend here instead would\n", " quietly hand the model an easier problem than the one it faces on real data.\n", " \"\"\"\n", " g = np.random.default_rng(seed)\n", " n = n_pre + n_post\n", " t = np.arange(n)\n", " walk = np.cumsum(g.normal(0.0, STEP_SD, n))\n", " level = BASE_CPA + DRIFT * t + walk + _dow(t)\n", " level[n_pre:] += true_effect\n", " dates = pd.date_range(\"2026-03-02\", periods=n, freq=\"D\")\n", " return pd.DataFrame({\n", " \"date\": dates,\n", " \"cpa\": np.round(level, 4),\n", " \"period\": [\"pre\"] * n_pre + [\"post\"] * n_post,\n", " })\n" ] }, { "cell_type": "code", "execution_count": 3, "id": "9612c0a8", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:43:06.353844Z", "iopub.status.busy": "2026-10-24T15:43:06.353466Z", "iopub.status.idle": "2026-10-24T15:43:06.549851Z", "shell.execute_reply": "2026-10-24T15:43:06.536161Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
datecpaperiod
02026-03-0233.1942pre
12026-03-0333.3524pre
22026-03-0433.6122pre
32026-03-0530.8811pre
42026-03-0630.1273pre
\n", "
" ], "text/plain": [ " date cpa period\n", "0 2026-03-02 33.1942 pre\n", "1 2026-03-03 33.3524 pre\n", "2 2026-03-04 33.6122 pre\n", "3 2026-03-05 30.8811 pre\n", "4 2026-03-06 30.1273 pre" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = generate_series()\n", "pre = df.loc[df.period == \"pre\", \"cpa\"].to_numpy()\n", "post = df.loc[df.period == \"post\", \"cpa\"].to_numpy()\n", "df.head()" ] }, { "cell_type": "markdown", "id": "67429096", "metadata": {}, "source": [ "## 2. What the before/after average says\n", "\n", "Average the days before, average the days after, subtract. This is what a\n", "dashboard shows you and what most people report." ] }, { "cell_type": "code", "execution_count": 4, "id": "b63e051d", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:43:06.581484Z", "iopub.status.busy": "2026-10-24T15:43:06.581074Z", "iopub.status.idle": "2026-10-24T15:43:06.627322Z", "shell.execute_reply": "2026-10-24T15:43:06.626398Z" } }, "outputs": [], "source": [ "def naive_prepost(df):\n", " \"\"\"What a marketer does by default: average before vs average after.\"\"\"\n", " pre = df.loc[df.period == \"pre\", \"cpa\"]\n", " post = df.loc[df.period == \"post\", \"cpa\"]\n", " return {\n", " \"pre_mean\": float(pre.mean()),\n", " \"post_mean\": float(post.mean()),\n", " \"diff\": float(post.mean() - pre.mean()),\n", " \"pct\": float((post.mean() - pre.mean()) / pre.mean() * 100.0),\n", " }\n" ] }, { "cell_type": "code", "execution_count": 5, "id": "d549ae45", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:43:06.631924Z", "iopub.status.busy": "2026-10-24T15:43:06.631564Z", "iopub.status.idle": "2026-10-24T15:43:06.652640Z", "shell.execute_reply": "2026-10-24T15:43:06.648279Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "before: 32.30 after: 36.30\n", "verdict: cost per lead moved +4.00 EUR (+12.4%)\n" ] } ], "source": [ "naive = naive_prepost(df)\n", "print(f\"before: {naive['pre_mean']:.2f} after: {naive['post_mean']:.2f}\")\n", "print(f\"verdict: cost per lead moved {naive['diff']:+.2f} EUR ({naive['pct']:+.1f}%)\")" ] }, { "cell_type": "markdown", "id": "bcfe42c7", "metadata": {}, "source": [ "Cost per lead went **up**. On this evidence the change failed and should be\n", "rolled back.\n", "\n", "Hold that thought." ] }, { "cell_type": "markdown", "id": "be41498a", "metadata": {}, "source": [ "## 3. The comparison the before/after average forgot\n", "\n", "The average of the days before is not what those days were heading towards. The\n", "metric was climbing. To judge the change you have to compare the days after\n", "against **where the series was going**, not against where it had been.\n", "\n", "That projection is the counterfactual. Building it takes three decisions, and the\n", "data makes all three." ] }, { "cell_type": "code", "execution_count": 6, "id": "754b8de4", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:43:06.657124Z", "iopub.status.busy": "2026-10-24T15:43:06.656788Z", "iopub.status.idle": "2026-10-24T15:43:06.680946Z", "shell.execute_reply": "2026-10-24T15:43:06.680184Z" } }, "outputs": [], "source": [ "def choose_d(pre_values, alpha=0.05):\n", " \"\"\"Augmented Dickey-Fuller on the raw pre-period picks the differencing order.\n", "\n", " p < 0.05 means the series already reverts to a stable level, so differencing\n", " it would only add noise; otherwise difference once. Standard Box-Jenkins\n", " practice.\n", " \"\"\"\n", " p_raw = adfuller(pre_values, autolag=\"AIC\")[1]\n", " return (0, float(p_raw)) if p_raw < alpha else (1, float(p_raw))\n", "\n", "def fit_pre(pre_values, seasonal_alpha=0.10):\n", " \"\"\"Fit SARIMA(1,d,0)(1,0,0,7); fall back to non-seasonal when the weekly term\n", " cannot be pinned.\n", "\n", " The seasonal test uses a 0.10 cutoff rather than the usual 0.05: with only a\n", " handful of complete weeks in the pre-period the weekly coefficient is\n", " imprecise, and a strict cutoff would throw away a real weekly pattern more\n", " often than it would keep a spurious one. The model learns whatever weekly\n", " shape the data carries - nothing about the business is pre-classified.\n", " \"\"\"\n", " d, adf_p = choose_d(pre_values)\n", " seasonal = SARIMAX(pre_values, order=(1, d, 0), seasonal_order=(1, 0, 0, 7),\n", " trend=\"c\", enforce_stationarity=False,\n", " enforce_invertibility=False).fit(disp=False)\n", " sar_ix = [i for i, n in enumerate(seasonal.param_names) if n.startswith(\"ar.S\")][0]\n", " sar_p = float(np.asarray(seasonal.pvalues)[sar_ix])\n", " base = {\"order\": (1, d, 0), \"adf_p\": adf_p, \"seasonal_ar_p\": sar_p}\n", " if sar_p < seasonal_alpha:\n", " return seasonal, {**base, \"seasonal_order\": (1, 0, 0, 7), \"seasonal\": True}\n", " plain = SARIMAX(pre_values, order=(1, d, 0), trend=\"c\",\n", " enforce_stationarity=False, enforce_invertibility=False).fit(disp=False)\n", " return plain, {**base, \"seasonal_order\": None, \"seasonal\": False}\n" ] }, { "cell_type": "markdown", "id": "06089601", "metadata": {}, "source": [ "### Does the model hold up on this series?\n", "\n", "Three checks, all standard. Residual independence (Ljung-Box, corrected for\n", "testing two lags at once), roughly normal residuals (Jarque-Bera), and steady\n", "noise across the window. If any of them fails hard, the honest move is to stop\n", "modelling and fall back to the plain comparison - which the notebook does." ] }, { "cell_type": "code", "execution_count": 7, "id": "314eba76", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:43:06.689318Z", "iopub.status.busy": "2026-10-24T15:43:06.688937Z", "iopub.status.idle": "2026-10-24T15:43:06.704691Z", "shell.execute_reply": "2026-10-24T15:43:06.700857Z" } }, "outputs": [], "source": [ "def diagnostics(res, spec):\n", " \"\"\"Ljung-Box (Bonferroni over lags above model_df), Jarque-Bera, H.\"\"\"\n", " model_df = spec[\"order\"][0] + (1 if spec[\"seasonal\"] else 0)\n", " lags = [l for l in (7, 14) if l > model_df]\n", " lb = acorr_ljungbox(res.resid, lags=lags, model_df=model_df, return_df=True)\n", " lb_min = float(lb[\"lb_pvalue\"].min())\n", " jb_stat, jb_p, skew, kurt = res.test_normality(\"jarquebera\")[0]\n", " h_stat, h_p = res.test_heteroskedasticity(\"breakvar\")[0][:2]\n", " cut = 0.05 / len(lags)\n", " return {\"ljung_box_min_p\": lb_min, \"ljung_box_lags\": lags, \"ljung_box_cut\": cut,\n", " \"ljung_box_pass\": bool(lb_min >= cut),\n", " \"jarque_bera_p\": float(jb_p), \"skewness\": float(skew),\n", " \"jarque_bera_pass\": bool(jb_p > 0.05),\n", " \"hetero_p\": float(h_p), \"hetero_pass\": bool(h_p > 0.05),\n", " \"model_df\": model_df}\n" ] }, { "cell_type": "code", "execution_count": 8, "id": "8ff10689", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:43:06.713596Z", "iopub.status.busy": "2026-10-24T15:43:06.708518Z", "iopub.status.idle": "2026-10-24T15:43:07.043862Z", "shell.execute_reply": "2026-10-24T15:43:07.036922Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "weekly pattern used: True (its p-value 0.0000)\n", "differencing: 1 (stationarity p-value 0.997)\n", "\n", " residuals independent pass\n", " residuals ~normal pass\n", " noise steady pass\n" ] } ], "source": [ "res, spec = fit_pre(pre)\n", "diag = diagnostics(res, spec)\n", "print(f\"weekly pattern used: {spec['seasonal']} (its p-value {spec['seasonal_ar_p']:.4f})\")\n", "print(f\"differencing: {spec['order'][1]} (stationarity p-value {spec['adf_p']:.3f})\")\n", "print()\n", "for name, ok in ((\"residuals independent\", diag[\"ljung_box_pass\"]),\n", " (\"residuals ~normal\", diag[\"jarque_bera_pass\"]),\n", " (\"noise steady\", diag[\"hetero_pass\"])):\n", " print(f\" {name:24s} {'pass' if ok else 'FAIL'}\")" ] }, { "cell_type": "markdown", "id": "7786db43", "metadata": {}, "source": [ "## 4. How far forward is it honest to look?\n", "\n", "The projection gets vaguer every day. There is a point where the comparison line\n", "is fuzzier than the metric's own day-to-day swings, and past that an extra day\n", "adds more uncertainty than it removes.\n", "\n", "The method finds that point itself. It is not a calendar rule and it is usually\n", "sooner than people expect." ] }, { "cell_type": "code", "execution_count": 9, "id": "fec16287", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:43:07.055931Z", "iopub.status.busy": "2026-10-24T15:43:07.055527Z", "iopub.status.idle": "2026-10-24T15:43:07.073581Z", "shell.execute_reply": "2026-10-24T15:43:07.071152Z" } }, "outputs": [], "source": [ "def read_horizon(res, n_post, pre_values):\n", " \"\"\"The method's own stopping rule: read while the per-day counterfactual\n", " standard error stays inside the pre-period's own spread.\"\"\"\n", " se = np.asarray(res.get_forecast(steps=n_post).se_mean)\n", " pre_sd = float(np.std(pre_values, ddof=1))\n", " over = se > pre_sd\n", " return (int(np.argmax(over)) if over.any() else int(n_post)), pre_sd\n" ] }, { "cell_type": "code", "execution_count": 10, "id": "194010ca", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:43:07.090423Z", "iopub.status.busy": "2026-10-24T15:43:07.089998Z", "iopub.status.idle": "2026-10-24T15:43:07.126875Z", "shell.execute_reply": "2026-10-24T15:43:07.125005Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "the metric's own normal swing: 2.72 EUR\n", "honest read window: 5 days\n" ] } ], "source": [ "horizon, pre_sd = read_horizon(res, len(post), pre)\n", "print(f\"the metric's own normal swing: {pre_sd:.2f} EUR\")\n", "print(f\"honest read window: {horizon} days\")" ] }, { "cell_type": "markdown", "id": "eaddfa31", "metadata": {}, "source": [ "## 5. The effect, and how sure you can be\n", "\n", "The effect is the average gap between what happened and the projection.\n", "\n", "Its uncertainty needs care. Each day's projection leans on the same fitted\n", "model, so the daily errors move together, and averaging the per-day error bars\n", "would quote a tighter range than the data supports. Instead: draw a set of model\n", "parameters, simulate a whole path from them, record the gap, and repeat. The\n", "spread of those gaps is the range." ] }, { "cell_type": "code", "execution_count": 11, "id": "fd8da9cd", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:43:07.147484Z", "iopub.status.busy": "2026-10-24T15:43:07.145479Z", "iopub.status.idle": "2026-10-24T15:43:07.184568Z", "shell.execute_reply": "2026-10-24T15:43:07.182358Z" } }, "outputs": [], "source": [ "def counterfactual(res, n_post):\n", " \"\"\"Project the fitted pre-period model across the post window.\"\"\"\n", " fc = res.get_forecast(steps=n_post)\n", " return np.asarray(fc.predicted_mean), fc.conf_int(alpha=0.05)\n", "\n", "def mean_effect_ci(res, post_actual, n_draws=2000, seed=SEED):\n", " \"\"\"Two-level draw: sample parameters from their joint distribution, simulate\n", " one path per draw, and read the interval off the distribution of mean\n", " effects. Averaging per-day forecast bounds understates the interval because\n", " daily forecast errors are correlated through shared model state.\"\"\"\n", " g = np.random.default_rng(seed)\n", " params = np.asarray(res.params)\n", " cov = np.asarray(res.cov_params())\n", " cov = (cov + cov.T) / 2.0\n", " draws = g.multivariate_normal(params, cov, size=n_draws)\n", " n_post = len(post_actual)\n", " actual_mean = float(np.mean(post_actual))\n", " samples, failed = [], 0\n", " for row in draws:\n", " try:\n", " r_i = res.model.smooth(row)\n", " path = np.asarray(r_i.simulate(nsimulations=n_post, repetitions=1,\n", " anchor=\"end\", random_state=g)).squeeze()\n", " m = float(np.mean(path))\n", " if not np.isfinite(m):\n", " failed += 1\n", " continue\n", " samples.append(actual_mean - m)\n", " except Exception:\n", " failed += 1\n", " samples = np.asarray(samples)\n", " return {\n", " \"mean_effect\": actual_mean - float(np.mean(counterfactual(res, n_post)[0])),\n", " \"ci_low\": float(np.quantile(samples, 0.025)),\n", " \"ci_high\": float(np.quantile(samples, 0.975)),\n", " \"n_draws\": int(n_draws),\n", " \"draws_used\": int(samples.size),\n", " \"draw_failure_share\": float(failed / n_draws),\n", " }\n" ] }, { "cell_type": "code", "execution_count": 12, "id": "e932a608", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:43:07.200059Z", "iopub.status.busy": "2026-10-24T15:43:07.199581Z", "iopub.status.idle": "2026-10-24T15:48:14.802016Z", "shell.execute_reply": "2026-10-24T15:48:14.800118Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "effect: -7.54 EUR per lead per day\n", "95% range: [-11.19, -3.81]\n", "range excludes zero: True\n", "\n", "before/after said: +4.00\n", "the change really did: -5.60 (we built it in, so we know)\n" ] } ], "source": [ "full = mean_effect_ci(res, post[:horizon], n_draws=10_000)\n", "print(f\"effect: {full['mean_effect']:+.2f} EUR per lead per day\")\n", "print(f\"95% range: [{full['ci_low']:+.2f}, {full['ci_high']:+.2f}]\")\n", "print(f\"range excludes zero: {full['ci_high'] < 0 or full['ci_low'] > 0}\")\n", "print()\n", "print(f\"before/after said: {naive['diff']:+.2f}\")\n", "print(f\"the change really did: {TRUE_EFFECT:+.2f} (we built it in, so we know)\")" ] }, { "cell_type": "markdown", "id": "1d9cce07", "metadata": {}, "source": [ "**The two methods disagree about the direction.** The before/after average says\n", "cost per lead went up. Against the counterfactual it went sharply down - and the\n", "true effect built into the data sits inside the range.\n", "\n", "Note what the range does *not* say. It is wide, and the point estimate overshoots\n", "the truth. A correctly built range is supposed to contain the answer, not to hit\n", "it." ] }, { "cell_type": "markdown", "id": "ce9d0d77", "metadata": {}, "source": [ "## 6. Could a quiet week have produced this?\n", "\n", "A fair objection: this metric jumps around. Maybe the model finds a \"break\"\n", "wherever you point it.\n", "\n", "So point it somewhere nothing happened. Take dates in the quiet period before the\n", "change, pretend each one was the change, and run the identical analysis. If those\n", "dates throw up breaks as strong as the real one, the analysis cannot tell your\n", "change apart from the metric's ordinary noise, and the honest verdict is weaker.\n", "\n", "This is a placebo test, and it is the check that separates a result from a\n", "coincidence." ] }, { "cell_type": "code", "execution_count": 13, "id": "69328304", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:48:14.840864Z", "iopub.status.busy": "2026-10-24T15:48:14.840368Z", "iopub.status.idle": "2026-10-24T15:48:14.890294Z", "shell.execute_reply": "2026-10-24T15:48:14.889325Z" } }, "outputs": [], "source": [ "def _effect_stat(pre_values, cut, horizon):\n", " \"\"\"Standardised effect if the intervention had happened at `cut`.\"\"\"\n", " train, held = pre_values[:cut], pre_values[cut:cut + horizon]\n", " if len(train) < 20 or len(held) < horizon:\n", " return None\n", " try:\n", " res, _ = fit_pre(train)\n", " fc = res.get_forecast(steps=horizon)\n", " eff = float(np.mean(held) - np.mean(np.asarray(fc.predicted_mean)))\n", " se = float(np.mean(np.asarray(fc.se_mean))) / np.sqrt(horizon)\n", " if not np.isfinite(eff) or not np.isfinite(se) or se <= 0:\n", " return None\n", " return abs(eff / se)\n", " except Exception:\n", " return None\n", "\n", "def falsification_sweep(pre_values, actual_stat, horizon, percentile=90):\n", " \"\"\"Ask the same question at dates where nothing happened. If the pre-period\n", " throws up 'effects' as strong as the real one, the design cannot pin the\n", " movement on the change (Baicker & Svoronos 2019; Eggers et al. 2024).\"\"\"\n", " n = len(pre_values)\n", " n_cand = int(np.clip(n // 7, 3, 12))\n", " cuts = np.linspace(int(n * 0.35), n - horizon - 1, n_cand).astype(int)\n", " stats = [s for s in (_effect_stat(pre_values, int(c), horizon) for c in cuts)\n", " if s is not None]\n", " if len(stats) < 3:\n", " return {\"ran\": False, \"reason\": \"too_few_usable_candidates\",\n", " \"candidates_placed\": len(cuts), \"candidates_usable\": len(stats)}\n", " cutoff = float(np.percentile(stats, percentile))\n", " return {\"ran\": True, \"candidates_placed\": len(cuts), \"candidates_usable\": len(stats),\n", " \"percentile\": percentile, \"cutoff\": cutoff,\n", " \"actual_stat\": float(actual_stat),\n", " \"strongest_placebo\": float(np.max(stats)),\n", " \"placebo_stats\": [float(s) for s in sorted(stats)],\n", " \"passes\": bool(actual_stat > cutoff)}\n", "\n", "def actual_effect_stat(res, post_actual, horizon):\n", " fc = res.get_forecast(steps=horizon)\n", " eff = float(np.mean(post_actual[:horizon]) - np.mean(np.asarray(fc.predicted_mean)))\n", " se = float(np.mean(np.asarray(fc.se_mean))) / np.sqrt(horizon)\n", " return abs(eff / se)\n" ] }, { "cell_type": "code", "execution_count": 14, "id": "1fccab64", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:48:14.903296Z", "iopub.status.busy": "2026-10-24T15:48:14.902599Z", "iopub.status.idle": "2026-10-24T15:48:16.874400Z", "shell.execute_reply": "2026-10-24T15:48:16.870119Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "quiet dates tested: 7\n", "strongest quiet-date break: 3.62\n", "the real change: 8.70\n", "clears the bar: True\n" ] } ], "source": [ "actual_stat = actual_effect_stat(res, post, horizon)\n", "fals = falsification_sweep(pre, actual_stat, horizon)\n", "print(f\"quiet dates tested: {fals['candidates_usable']}\")\n", "print(f\"strongest quiet-date break: {fals['strongest_placebo']:.2f}\")\n", "print(f\"the real change: {fals['actual_stat']:.2f}\")\n", "print(f\"clears the bar: {fals['passes']}\")" ] }, { "cell_type": "markdown", "id": "94cda034", "metadata": {}, "source": [ "## 7. The pictures" ] }, { "cell_type": "code", "execution_count": 15, "id": "d444a760", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:48:16.885816Z", "iopub.status.busy": "2026-10-24T15:48:16.885018Z", "iopub.status.idle": "2026-10-24T15:48:16.935168Z", "shell.execute_reply": "2026-10-24T15:48:16.934393Z" } }, "outputs": [], "source": [ "def _style(ax):\n", " ax.set_facecolor(PAPER)\n", " for s in (\"top\", \"right\"):\n", " ax.spines[s].set_visible(False)\n", " for s in (\"left\", \"bottom\"):\n", " ax.spines[s].set_color(LINE)\n", " ax.tick_params(colors=MUTED, labelsize=9)\n", " ax.grid(True, color=LINE, lw=0.8, alpha=0.7)\n", " ax.set_axisbelow(True)\n", "\n", "def _save(fig, name):\n", " fig.savefig(f\"{FIGDIR}/{name}\", dpi=150, bbox_inches=\"tight\", facecolor=PAPER)\n", " plt.close(fig)\n", "\n", "def fig_series(df, res, horizon, n_pre):\n", " \"\"\"Project the counterfactual across the WHOLE post window, not just the read\n", " window. The gap has to be visibly sustained or a reader reads the metric's\n", " later wandering as the change wearing off; and the band's growth beyond the\n", " read window is the visual argument for closing the window where we close it.\"\"\"\n", " fig, ax = plt.subplots(figsize=(9.2, 4.8), facecolor=PAPER); _style(ax)\n", " t = np.arange(len(df))\n", " n_post = len(df) - n_pre\n", " fc = res.get_forecast(steps=n_post)\n", " cf = np.asarray(fc.predicted_mean); ci = np.asarray(fc.conf_int(alpha=0.05))\n", " tp = np.arange(n_pre, len(df))\n", " # ANCHOR THE PROJECTION TO THE LAST OBSERVED DAY. Drawn from the first forecast day alone, the\n", " # dashed line starts wherever the model's one-step-ahead prediction sits -- three euros above the\n", " # last real point here -- and reads as an unexplained jump rather than as a continuation. The\n", " # forecast genuinely does start there; what was missing is the segment joining it to the data it\n", " # was fitted on, which is the standard way a forecast is drawn and how the video shows it.\n", " anchor_t = n_pre - 1\n", " anchor_y = float(np.asarray(df.cpa)[anchor_t])\n", " tpa = np.concatenate([[anchor_t], tp])\n", " cfa = np.concatenate([[anchor_y], cf])\n", " ci_lo = np.concatenate([[anchor_y], ci[:, 0]])\n", " ci_hi = np.concatenate([[anchor_y], ci[:, 1]])\n", " ax.fill_between(tpa[:horizon + 1], ci_lo[:horizon + 1], ci_hi[:horizon + 1],\n", " color=WARM, alpha=0.16, lw=0)\n", " # One colour across both periods. The before/after split is a TIME distinction,\n", " # already carried by the change rule and the shaded window - spending a colour\n", " # slot on it would double-encode, and the near-black it used reads as grey rather\n", " # than as an identity hue. Two slots: observed against projected.\n", " ax.plot(t, df.cpa, color=BLUE, lw=1.8, label=\"Cost per lead, observed\")\n", " ax.plot(tpa, cfa, color=WARM, lw=1.5, ls=\"--\", alpha=0.45)\n", " ax.plot(tpa[:horizon + 1], cfa[:horizon + 1], color=WARM, lw=2.2, ls=\"--\",\n", " label=\"What it would have done anyway\")\n", " ax.axvspan(n_pre - 0.5, n_pre - 0.5 + horizon, color=GREEN, alpha=0.08, lw=0)\n", " ax.axvline(n_pre - 0.5, color=MUTED, lw=1.1, ls=\":\")\n", " lo, hi = ax.get_ylim(); ax.set_ylim(lo, hi + (hi - lo) * 0.10)\n", " ax.annotate(\"target CPA cut\", xy=(n_pre - 0.5, hi), xytext=(-6, 6),\n", " textcoords=\"offset points\", ha=\"right\", color=MUTED, fontsize=9)\n", " ax.annotate(f\"the {horizon} days the estimate reads\",\n", " xy=(n_pre - 0.5 + horizon / 2.0, lo), xytext=(0, 6),\n", " textcoords=\"offset points\", ha=\"center\", color=GREEN, fontsize=9)\n", " ax.set_xlabel(\"Day\", color=MUTED, fontsize=9)\n", " ax.set_ylabel(\"Cost per lead (EUR)\", color=MUTED, fontsize=9)\n", " ax.set_title(\"The line the change was measured against\",\n", " color=INK, fontsize=12, loc=\"left\", pad=10)\n", " ax.legend(frameon=False, fontsize=9, labelcolor=MUTED, loc=\"upper left\")\n", " _save(fig, \"fig_series.png\")\n", "\n", "def fig_naive_vs_its(naive, full, true_effect):\n", " \"\"\"Point against point. The interval is a distinct whisker, never the mark.\"\"\"\n", " fig, ax = plt.subplots(figsize=(8.0, 3.4), facecolor=PAPER); _style(ax)\n", " rows = [(\"Before/after average\", naive[\"diff\"], None, None, BLUE),\n", " (\"Against the counterfactual\", full[\"mean_effect\"],\n", " full[\"ci_low\"], full[\"ci_high\"], GREEN)]\n", " for i, (lab, val, lo, hi, col) in enumerate(rows):\n", " if lo is not None:\n", " ax.plot([lo, hi], [i, i], color=col, lw=2.0, alpha=0.45,\n", " solid_capstyle=\"butt\")\n", " ax.plot([lo, lo], [i - .09, i + .09], color=col, lw=2.0)\n", " ax.plot([hi, hi], [i - .09, i + .09], color=col, lw=2.0)\n", " ax.plot([val], [i], \"o\", color=col, ms=9, zorder=3)\n", " ax.annotate(f\"{val:+.2f}\", xy=(val, i), xytext=(0, 12),\n", " textcoords=\"offset points\", ha=\"center\",\n", " color=col, fontsize=10, fontweight=\"bold\")\n", " ax.axvline(0, color=MUTED, lw=1.0)\n", " ax.axvline(true_effect, color=MUTED, lw=1.2, ls=\"--\")\n", " ax.annotate(\"what the change really did\", xy=(true_effect, 1.42),\n", " xytext=(6, 0), textcoords=\"offset points\",\n", " color=MUTED, fontsize=9)\n", " ax.set_yticks([0, 1]); ax.set_yticklabels([r[0] for r in rows], fontsize=10)\n", " ax.set_ylim(-0.55, 1.7)\n", " ax.margins(x=0.10)\n", " ax.set_xlabel(\"Change in cost per lead (EUR/day)\", color=MUTED, fontsize=9)\n", " ax.set_title(\"Same data, opposite verdicts\", color=INK, fontsize=12,\n", " loc=\"left\", pad=10)\n", " _save(fig, \"fig_naive_vs_its.png\")\n" ] }, { "cell_type": "code", "execution_count": 16, "id": "aeb14e99", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:48:16.942795Z", "iopub.status.busy": "2026-10-24T15:48:16.942140Z", "iopub.status.idle": "2026-10-24T15:48:16.975320Z", "shell.execute_reply": "2026-10-24T15:48:16.974463Z" } }, "outputs": [], "source": [ "def fig_horizon(res, n_post, pre_values, horizon):\n", " \"\"\"Why the read window closes when it does.\"\"\"\n", " se = np.asarray(res.get_forecast(steps=n_post).se_mean)\n", " pre_sd = float(np.std(pre_values, ddof=1))\n", " fig, ax = plt.subplots(figsize=(8.4, 3.8), facecolor=PAPER); _style(ax)\n", " days = np.arange(1, n_post + 1)\n", " ax.plot(days, se, color=WARM, lw=2.0, label=\"Uncertainty in the comparison line\")\n", " ax.axhline(pre_sd, color=INK, lw=1.4, ls=\"--\",\n", " label=\"How much the metric normally moves\")\n", " ax.axvspan(0.5, horizon + 0.5, color=GREEN, alpha=0.08, lw=0)\n", " ax.annotate(f\"read {horizon} days\", xy=(horizon / 2.0 + 0.5, se.min()),\n", " xytext=(0, -2), textcoords=\"offset points\",\n", " ha=\"center\", va=\"top\", color=GREEN, fontsize=9)\n", " ax.annotate(\"past here the comparison line is\\nvaguer than the metric's own swings\",\n", " xy=(horizon + 1.2, pre_sd), xytext=(10, 14),\n", " textcoords=\"offset points\", color=MUTED, fontsize=9)\n", " ax.set_ylim(bottom=se.min() - (se.max() - se.min()) * 0.14)\n", " ax.set_xlabel(\"Days after the change\", color=MUTED, fontsize=9)\n", " ax.set_ylabel(\"EUR\", color=MUTED, fontsize=9)\n", " ax.set_title(\"Why the window is days, not weeks\", color=INK, fontsize=12,\n", " loc=\"left\", pad=10)\n", " ax.legend(frameon=False, fontsize=9, labelcolor=MUTED, loc=\"upper left\")\n", " _save(fig, \"fig_horizon.png\")\n", "\n", "def fig_falsification(fs):\n", " \"\"\"The real change against the same question asked on quiet dates. Show the\n", " placebos themselves, not just the cutoff - the point is the reader seeing how\n", " far outside the quiet-period spread the real change sits.\"\"\"\n", " fig, ax = plt.subplots(figsize=(8.4, 3.2), facecolor=PAPER); _style(ax)\n", " stats = fs[\"placebo_stats\"]\n", " ax.plot(stats, [0] * len(stats), \"o\", color=MUTED, ms=9, alpha=0.55,\n", " label=f\"{len(stats)} quiet dates, same question asked\")\n", " ax.plot([fs[\"actual_stat\"]], [0], \"o\", color=GREEN, ms=14, zorder=3,\n", " label=\"the day of the real change\")\n", " ax.annotate(f\"{fs['actual_stat']:.1f}\", xy=(fs[\"actual_stat\"], 0),\n", " xytext=(0, 16), textcoords=\"offset points\", ha=\"center\",\n", " color=GREEN, fontsize=11, fontweight=\"bold\")\n", " ax.annotate(\"every quiet date lands below 4\", xy=(np.median(stats), 0),\n", " xytext=(0, -30), textcoords=\"offset points\", ha=\"center\",\n", " color=MUTED, fontsize=9)\n", " ax.set_yticks([]); ax.set_ylim(-1.0, 1.0)\n", " ax.set_xlim(0, fs[\"actual_stat\"] * 1.15)\n", " ax.spines[\"left\"].set_visible(False)\n", " ax.grid(True, axis=\"x\", color=LINE, lw=0.8, alpha=0.7)\n", " ax.set_xlabel(\"Strength of the break (higher = harder to explain by chance)\",\n", " color=MUTED, fontsize=9)\n", " ax.set_title(\"Could a quiet week have produced this?\", color=INK, fontsize=12,\n", " loc=\"left\", pad=10)\n", " ax.legend(frameon=False, fontsize=9, labelcolor=MUTED, loc=\"upper left\")\n", " _save(fig, \"fig_falsification.png\")\n" ] }, { "cell_type": "code", "execution_count": 17, "id": "2038e82f", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:48:16.985923Z", "iopub.status.busy": "2026-10-24T15:48:16.985605Z", "iopub.status.idle": "2026-10-24T15:48:19.556664Z", "shell.execute_reply": "2026-10-24T15:48:19.555582Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "figures written\n" ] } ], "source": [ "fig_series(df, res, horizon, N_PRE)\n", "fig_naive_vs_its(naive, full, TRUE_EFFECT)\n", "fig_horizon(res, len(post), pre, horizon)\n", "fig_falsification(fals)\n", "print(\"figures written\")" ] }, { "cell_type": "markdown", "id": "ddf17aeb", "metadata": {}, "source": [ "## 8. The simplified version - and what it costs\n", "\n", "The full model cannot run in a spreadsheet or a browser, so the interactive\n", "version of this post uses a simpler counterfactual: carry the level the series\n", "had just before the change forward at the pace it was already moving.\n", "\n", "It is worth seeing what that simplification costs, because the honest answer is\n", "\"precision, not direction\"." ] }, { "cell_type": "code", "execution_count": 18, "id": "70d31655", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:48:19.571134Z", "iopub.status.busy": "2026-10-24T15:48:19.570709Z", "iopub.status.idle": "2026-10-24T15:48:19.594615Z", "shell.execute_reply": "2026-10-24T15:48:19.590618Z" } }, "outputs": [], "source": [ "ANCHOR_DAYS = 7 # trailing window the level is read from (one full week)\n", "\n", "def linear_its(pre_values, post_values, horizon=None, anchor_days=ANCHOR_DAYS):\n", " \"\"\"Drift-method counterfactual, projected across the read window.\"\"\"\n", " y = np.asarray(pre_values, dtype=float)\n", " n = len(y)\n", " h = int(horizon or len(post_values))\n", " drift = float((y[-1] - y[0]) / (n - 1))\n", " k = min(anchor_days, n)\n", " anchor = float(y[-k:].mean()) # level at the centre of the window\n", " offset = (k - 1) / 2.0 # days from that centre to the last day\n", " cf = np.array([anchor + drift * (offset + j) for j in range(1, h + 1)])\n", " steps = np.diff(y)\n", " s_d = float(np.std(steps, ddof=1))\n", " post = np.asarray(post_values, dtype=float)[:h]\n", " effect = float(post.mean() - cf.mean())\n", " var = (s_d ** 2) * (h * (h + 1) * (2 * h + 1) / 6.0) / (h ** 2)\n", " se = float(np.sqrt(var))\n", " return {\"drift\": drift, \"anchor\": anchor, \"step_sd\": s_d, \"horizon\": h,\n", " \"counterfactual_mean\": float(cf.mean()),\n", " \"post_mean\": float(post.mean()),\n", " \"effect\": effect, \"se\": se,\n", " \"ci_low\": effect - 1.96 * se, \"ci_high\": effect + 1.96 * se,\n", " \"counterfactual\": [float(v) for v in cf]}\n" ] }, { "cell_type": "code", "execution_count": 19, "id": "3734d4d0", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:48:19.600513Z", "iopub.status.busy": "2026-10-24T15:48:19.598740Z", "iopub.status.idle": "2026-10-24T15:48:19.614897Z", "shell.execute_reply": "2026-10-24T15:48:19.614080Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "full model: -7.54 [-11.19, -3.81]\n", "simplified: -3.54 [-8.60, +1.53]\n", "\n", "true effect -5.60 inside full model range: True\n", "true effect -5.60 inside simplified range: True\n" ] } ], "source": [ "lin = linear_its(pre, post, horizon)\n", "print(f\"full model: {full['mean_effect']:+.2f} [{full['ci_low']:+.2f}, {full['ci_high']:+.2f}]\")\n", "print(f\"simplified: {lin['effect']:+.2f} [{lin['ci_low']:+.2f}, {lin['ci_high']:+.2f}]\")\n", "print()\n", "print(f\"true effect {TRUE_EFFECT:+.2f} inside full model range: \"\n", " f\"{full['ci_low'] <= TRUE_EFFECT <= full['ci_high']}\")\n", "print(f\"true effect {TRUE_EFFECT:+.2f} inside simplified range: \"\n", " f\"{lin['ci_low'] <= TRUE_EFFECT <= lin['ci_high']}\")" ] }, { "cell_type": "markdown", "id": "dc058aa2", "metadata": {}, "source": [ "Same direction, wider range, both containing the truth. The simplified version is\n", "a fair thing to play with; it is not the thing to publish a number from." ] }, { "cell_type": "markdown", "id": "cf0c870c", "metadata": {}, "source": [ "## 9. Writing the numbers out\n", "\n", "Every number the article quotes is written here and read from these files.\n", "Nothing in the article is typed by hand." ] }, { "cell_type": "code", "execution_count": 20, "id": "4ebbf899", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:48:19.633588Z", "iopub.status.busy": "2026-10-24T15:48:19.632926Z", "iopub.status.idle": "2026-10-24T15:48:19.654203Z", "shell.execute_reply": "2026-10-24T15:48:19.653365Z" } }, "outputs": [], "source": [ "TOLERANCE = 1e-4\n", "PINNED = [\n", " {\"label\": \"the article's own read\", \"read_window\": None},\n", " {\"label\": \"a single week\", \"read_window\": 7},\n", " {\"label\": \"a fortnight\", \"read_window\": 14},\n", " {\"label\": \"one day after\", \"read_window\": 1},\n", "]\n", "\n", "def emit_checks(pre, post, horizon, cf_path, sweep, path=\"data/checks.json\"):\n", " \"\"\"Pin what the widget must reproduce: the effect at every readable window.\n", "\n", " This file used to pin a SIMPLIFIED drift engine that the widget ran client-side instead of the\n", " seasonal model behind the article. Same series, different estimator -- and the two disagreed on\n", " the one thing the post is about, the widget showing a range that crossed zero on the very\n", " scenario the article calls its own. The widget now computes the article's own quantity, the mean\n", " gap between what happened and the projection, so the check below is a check on arithmetic the\n", " reader can follow rather than on a second model nobody mentions.\n", " \"\"\"\n", " sets = []\n", " for spec in PINNED:\n", " h = spec[\"read_window\"] or horizon\n", " row = next(r for r in sweep if r[\"days\"] == h)\n", " sets.append({\n", " \"label\": spec[\"label\"], \"read_window\": h,\n", " \"expected\": {\"effect\": row[\"effect\"],\n", " \"ci_low\": row[\"ci_low\"], \"ci_high\": row[\"ci_high\"],\n", " \"post_mean\": round(float(np.mean(post[:h])), 6),\n", " \"counterfactual_mean\": round(float(np.mean(cf_path[:h])), 6)},\n", " })\n", " payload = {\n", " \"note\": (\"Pinned outputs of the SAME estimator the article quotes, read at four windows. \"\n", " \"The widget recomputes the effect as the mean gap between the observed days and \"\n", " \"the projection below, and must reproduce every set here or its self-check fails. \"\n", " \"The interval is not recomputed client-side and is not meant to be: daily forecast \"\n", " \"errors are correlated, so an aggregate interval cannot be rebuilt from per-day \"\n", " \"bounds -- which is the article's own point.\"),\n", " \"tolerance\": TOLERANCE,\n", " \"series\": {\"cpa\": [round(float(v), 4) for v in np.concatenate([pre, post])],\n", " \"n_pre\": int(len(pre))},\n", " \"counterfactual\": [round(float(v), 4) for v in cf_path],\n", " \"window_sweep\": sweep,\n", " \"sets\": sets,\n", " }\n", " with open(path, \"w\", encoding=\"utf-8\") as f:\n", " json.dump(payload, f, indent=2)\n", " return payload\n" ] }, { "cell_type": "code", "execution_count": 21, "id": "2c287d88", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:48:19.664199Z", "iopub.status.busy": "2026-10-24T15:48:19.663801Z", "iopub.status.idle": "2026-10-24T15:48:19.710068Z", "shell.execute_reply": "2026-10-24T15:48:19.709094Z" } }, "outputs": [], "source": [ "def _implied_drift(res):\n", " \"\"\"The long-run daily drift the fitted model projects: c / ((1 - phi1) * (1 - Phi1)).\"\"\"\n", " p = dict(zip(res.param_names, np.asarray(res.params, dtype=float)))\n", " phi = p.get(\"ar.L1\", 0.0)\n", " Phi = p.get(\"ar.S.L7\", 0.0)\n", " return p.get(\"intercept\", 0.0) / ((1.0 - phi) * (1.0 - Phi))\n", "\n", "\n", "def emit_worked_example(df, res, spec, diag, naive, full, lin, fs, horizon,\n", " pre_sd, sweep, path=\"data/worked_example.json\"):\n", " \"\"\"Every headline number the article prose quotes. Prose reads from here;\n", " nothing is retyped.\"\"\"\n", " # The model's own forecast across the post window, with its band. Emitted because the\n", " # explainer video needs to DRAW the projection, and a drawn approximation of a fitted\n", " # line is exactly the defect review caught on the trend slide: a line shown over data\n", " # has to be the line the model actually produced.\n", " cf_path, cf_ci = counterfactual(res, N_POST)\n", " # The SAME estimator the headline number comes from, read at every window a reader can choose.\n", " # The widget used to run a simplified drift engine instead, which put a range crossing zero on\n", " # the page three paragraphs under prose saying the range excludes it. Identical call, identical\n", " # seed, so the sweep AT the article's own horizon returns the article's own numbers rather than\n", " # something merely close to them.\n", " payload = {\n", " \"note\": (\"Synthetic data, labelled as such. Generated by this notebook \"\n", " \"under a single fixed seed, so every figure and every number in \"\n", " \"the article regenerates identically.\"),\n", " \"data_label\": \"synthetic\",\n", " \"seed\": SEED,\n", " \"series\": {\"n_pre\": int(N_PRE), \"n_post\": int(N_POST),\n", " \"pre_sd\": round(float(pre_sd), 4),\n", " \"start_date\": str(df.date.iloc[0].date()),\n", " \"change_date\": str(df.date.iloc[N_PRE].date())},\n", " \"ground_truth\": {\"true_effect_eur_per_day\": float(TRUE_EFFECT),\n", " \"drift_eur_per_day\": float(DRIFT)},\n", " \"naive\": {k: round(float(v), 4) for k, v in naive.items()},\n", " # The fitted parameters, and the drift the model actually PROJECTS. The constant is not the\n", " # drift: with a seasonal weight near one on the differenced series, the long-run drift is\n", " # c / ((1 - phi) * (1 - Phi)), several times the constant. The article once read the\n", " # constant as \"about 9 cents a day\" while the projection climbed at over fifty; emitting\n", " # both is what lets the prose say which is which.\n", " \"model\": {\"seasonal\": bool(spec[\"seasonal\"]),\n", " \"order\": list(spec[\"order\"]),\n", " \"adf_p\": round(float(spec[\"adf_p\"]), 6),\n", " \"seasonal_ar_p\": round(float(spec[\"seasonal_ar_p\"]), 6),\n", " \"params\": {k: round(float(v), 4)\n", " for k, v in zip(res.param_names, np.asarray(res.params))},\n", " \"implied_drift_eur_per_day\": round(float(_implied_drift(res)), 4),\n", " \"projected_climb_eur_per_day\": round(float((cf_path[-1] - cf_path[0]) / (N_POST - 1)), 4)},\n", " \"diagnostics\": {k: (round(float(v), 6) if isinstance(v, float) else v)\n", " for k, v in diag.items()},\n", " \"read_window\": {\"days\": int(horizon),\n", " \"rule\": (\"read while the comparison line's uncertainty stays \"\n", " \"inside the metric's own normal swing\")},\n", " \"its\": {\"mean_effect\": round(float(full[\"mean_effect\"]), 4),\n", " \"ci_low\": round(float(full[\"ci_low\"]), 4),\n", " \"ci_high\": round(float(full[\"ci_high\"]), 4),\n", " \"draws\": int(full[\"n_draws\"]),\n", " \"draws_used\": int(full[\"draws_used\"]),\n", " \"draw_failure_share\": round(float(full[\"draw_failure_share\"]), 6),\n", " \"excludes_zero\": bool(full[\"ci_high\"] < 0 or full[\"ci_low\"] > 0)},\n", " \"simplified\": {k: round(float(lin[k]), 4) for k in\n", " (\"drift\", \"anchor\", \"step_sd\", \"effect\", \"se\",\n", " \"ci_low\", \"ci_high\")},\n", " \"falsification\": {k: (round(float(v), 4) if isinstance(v, float) else v)\n", " for k, v in fs.items()},\n", " \"window_sweep\": {\n", " \"note\": (\"The effect and its 95% interval at every readable window, from the same \"\n", " \"estimator and the same seed as the headline number above. The widget reads \"\n", " \"this, so the page cannot quote two different answers for one account.\"),\n", " \"draws\": int(SWEEP_DRAWS),\n", " \"rows\": sweep,\n", " },\n", " \"projection\": {\n", " \"note\": (\"The fitted pre-period model run forward across the post window: what \"\n", " \"the metric was heading for had nothing changed, with its 95% band. \"\n", " \"One entry per post day, in order.\"),\n", " \"counterfactual\": [round(float(v), 4) for v in cf_path],\n", " \"ci_low\": [round(float(v), 4) for v in np.asarray(cf_ci)[:, 0]],\n", " \"ci_high\": [round(float(v), 4) for v in np.asarray(cf_ci)[:, 1]],\n", " },\n", " \"covers_truth\": {\n", " \"full_model\": bool(full[\"ci_low\"] <= TRUE_EFFECT <= full[\"ci_high\"]),\n", " \"simplified\": bool(lin[\"ci_low\"] <= TRUE_EFFECT <= lin[\"ci_high\"]),\n", " },\n", " }\n", " with open(path, \"w\", encoding=\"utf-8\") as f:\n", " json.dump(payload, f, indent=2)\n", " return payload\n" ] }, { "cell_type": "code", "execution_count": 22, "id": "c9ac2b8f", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T15:48:19.719205Z", "iopub.status.busy": "2026-10-24T15:48:19.718800Z", "iopub.status.idle": "2026-10-24T16:55:33.962061Z", "shell.execute_reply": "2026-10-24T16:55:33.959381Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "data/checks.json 4 pinned sets, 21 swept windows\n", "data/worked_example.json 15 blocks\n" ] } ], "source": [ "cf_path_emit, _cf_ci_emit = counterfactual(res, N_POST)\n", "post_actual = np.asarray(df.cpa)[N_PRE:]\n", "# ONE sweep, both files. Identical call and seed to the headline interval, so the row at the\n", "# article's own horizon IS the article's own numbers rather than something close to them.\n", "sweep = []\n", "for _h in range(1, N_POST + 1):\n", " _r = mean_effect_ci(res, post_actual[:_h], n_draws=SWEEP_DRAWS)\n", " sweep.append({\"days\": _h,\n", " \"effect\": round(float(_r[\"mean_effect\"]), 4),\n", " \"ci_low\": round(float(_r[\"ci_low\"]), 4),\n", " \"ci_high\": round(float(_r[\"ci_high\"]), 4)})\n", "\n", "checks = emit_checks(pre, post, horizon, cf_path_emit, sweep)\n", "worked = emit_worked_example(df, res, spec, diag, naive, full, lin, fals, horizon, pre_sd, sweep)\n", "print(\"data/checks.json \", len(checks[\"sets\"]), \"pinned sets,\", len(sweep), \"swept windows\")\n", "print(\"data/worked_example.json \", len(worked), \"blocks\")\n" ] }, { "cell_type": "markdown", "id": "a2c0ba7e", "metadata": {}, "source": [ "---\n", "\n", "# Part two: your own data\n", "\n", "Everything below runs on your machine. Your file is read from disk, analysed in\n", "this process, and nothing is sent anywhere.\n", "\n", "You need two things: a daily export covering a good stretch **before** your\n", "change, and the date the change actually went live." ] }, { "cell_type": "code", "execution_count": 23, "id": "4026d0e6", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T16:55:34.018006Z", "iopub.status.busy": "2026-10-24T16:55:34.012825Z", "iopub.status.idle": "2026-10-24T16:55:34.798601Z", "shell.execute_reply": "2026-10-24T16:55:34.795019Z" } }, "outputs": [], "source": [ "# ---- export reader (shared across every Stochastic Strata notebook; do not edit here) ----------\n", "# One reader for the file you bring. Ad platforms and analytics tools do not hand over clean tables:\n", "# a Google Ads download opens with a title line and a date-range line and closes with \"Total:\" rows,\n", "# a GA4 download opens with \"#\" lines and writes dates as 20261019, a European download separates\n", "# with semicolons and writes 1.234,56, and Excel turns numbers into \"1,234.56\" or \"€12.30\". This\n", "# reader finds the real header row, sets the title, comment and total rows aside, turns formatted\n", "# numbers back into numbers, and reads dates in the platforms' own forms. Where a file is genuinely\n", "# ambiguous - 06/07/2026 could be the 6th of July or the 7th of June - it stops and asks rather than\n", "# guessing, because a guessed date silently moves a day to the wrong weekday.\n", "#\n", "# table, notes = read_export(\"my_export.csv\", columns={\"date\": [\"day\", \"date\"], ...})\n", "# for note in notes: print(note)\n", "#\n", "# `columns` maps the names this notebook uses to the names your file might use (case and spacing do\n", "# not matter, and a currency suffix such as \"(EUR)\" is ignored). `date_columns` are read as dates,\n", "# `id_columns` are kept as text. Everything else that reads as numbers becomes numbers; a share\n", "# written \"49.41%\" becomes 0.4941. A share the platform shows only as \"< 10%\" or \"> 90%\" is read as\n", "# 0.10 or 0.90 and the direction is kept in a column ending \"_bound\", because it is a limit, not a\n", "# measurement. Every row set aside, and why, comes back in `notes`.\n", "import csv as _csv\n", "import io as _io\n", "import re as _re\n", "from pathlib import Path as _Path\n", "\n", "import numpy as _np\n", "import pandas as _pd\n", "\n", "\n", "class ExportReadError(ValueError):\n", " \"\"\"The file cannot be read without guessing; the message says what to change.\"\"\"\n", "\n", "\n", "_TOTAL_ROW = _re.compile(r\"^\\s*(grand\\s+)?totals?\\b\", _re.I)\n", "_CURRENCY = _re.compile(r\"[€$£¥₹]|\\b(EUR|USD|GBP|CHF|SEK|NOK|DKK|PLN|AUD|CAD)\\b\", _re.I)\n", "_EMPTY = {\"\", \"--\", \"-\", \"—\", \"–\", \"n/a\", \"na\", \"nan\", \"none\", \"null\", \"(not set)\"}\n", "_MONTHS = {m: i for i, m in enumerate(\n", " [\"jan\", \"feb\", \"mar\", \"apr\", \"may\", \"jun\", \"jul\", \"aug\", \"sep\", \"oct\", \"nov\", \"dec\"], 1)}\n", "\n", "\n", "def _norm(name):\n", " name = _re.sub(r\"\\((?:[A-Z]{3}|[€$£])\\)\", \"\", str(name))\n", " return _re.sub(r\"[\\s_\\-./]+\", \" \", name).strip().lower()\n", "\n", "\n", "def _decode(raw):\n", " if raw[:2] in (b\"\\xff\\xfe\", b\"\\xfe\\xff\"):\n", " return raw.decode(\"utf-16\")\n", " for encoding in (\"utf-8-sig\", \"cp1252\", \"latin-1\"):\n", " try:\n", " return raw.decode(encoding)\n", " except UnicodeDecodeError:\n", " continue\n", " return raw.decode(\"utf-8\", errors=\"replace\")\n", "\n", "\n", "def _rows_from(source):\n", " \"\"\"Every row of the file as a list of strings, whatever the container.\"\"\"\n", " if isinstance(source, _pd.DataFrame):\n", " return [list(map(str, source.columns))] + source.astype(str).values.tolist(), \",\"\n", " path = _Path(source)\n", " if not path.exists():\n", " raise ExportReadError(f\"There is no file at {path}. Check the name and the folder.\")\n", " if path.suffix.lower() in (\".xlsx\", \".xlsm\", \".xls\"):\n", " sheet = _pd.read_excel(path, header=None, dtype=str).fillna(\"\")\n", " return sheet.values.tolist(), \",\"\n", " text = _decode(path.read_bytes())\n", " lines = [ln for ln in text.splitlines() if not ln.lstrip().startswith(\"#\")]\n", " counts = {d: sum(ln.count(d) for ln in lines[:60]) for d in (\",\", \";\", \"\\t\")}\n", " delimiter = max(counts, key=counts.get) if any(counts.values()) else \",\"\n", " return [row for row in _csv.reader(_io.StringIO(\"\\n\".join(lines)), delimiter=delimiter)], delimiter\n", "\n", "\n", "def _find_header(rows, wanted):\n", " \"\"\"The first row that names the columns: the one matching most expected names, else the first\n", " row as wide as the data below it that is mostly words rather than numbers.\"\"\"\n", " widths = [len([c for c in r if str(c).strip()]) for r in rows]\n", " if wanted:\n", " best, best_hits = None, 0\n", " for i, row in enumerate(rows[:40]):\n", " names = {_norm(c) for c in row if str(c).strip()}\n", " hits = sum(1 for options in wanted.values() if names & set(options))\n", " if hits > best_hits:\n", " best, best_hits = i, hits\n", " if best is not None:\n", " return best\n", " body = [w for w in widths if w > 1]\n", " width = max(set(body), key=body.count) if body else 1\n", " for i, row in enumerate(rows[:40]):\n", " cells = [str(c).strip() for c in row if str(c).strip()]\n", " if len(cells) >= max(2, width - 1) and sum(_looks_number(c) for c in cells) <= len(cells) / 2:\n", " return i\n", " return 0\n", "\n", "\n", "def _looks_number(text):\n", " if _re.fullmatch(r\"\\d{1,4}[./\\-]\\d{1,2}[./\\-]\\d{1,4}\", str(text).strip()):\n", " return False # a date such as 22.09.2026, not a number with separators\n", " return bool(_re.fullmatch(r\"[<>]?\\s*[-+(]?[€$£]?\\s*[\\d.,\\s']+\\s*%?\\)?\", str(text).strip()))\n", "\n", "\n", "def _decimal_comma(values):\n", " \"\"\"True when these values write 1.234,56 (a comma followed by one or two final digits, or dots\n", " grouping thousands with a comma after them), False when they write 1,234.56, None when they do\n", " not say (whole numbers only, or a lone '1.340').\"\"\"\n", " comma_votes = dot_votes = 0\n", " for v in values:\n", " s = _re.sub(r\"[^\\d.,]\", \"\", v)\n", " if \",\" in s and \".\" in s:\n", " comma_votes += s.rfind(\",\") > s.rfind(\".\")\n", " dot_votes += s.rfind(\".\") > s.rfind(\",\")\n", " elif _re.search(r\",\\d{1,2}$\", s):\n", " comma_votes += 1\n", " elif _re.search(r\"\\.\\d{1,2}$\", s) or _re.search(r\"\\.\\d{4,}$\", s):\n", " dot_votes += 1\n", " if comma_votes == dot_votes:\n", " return None\n", " return comma_votes > dot_votes\n", "\n", "\n", "def _to_number(text, comma_decimal):\n", " s = str(text).strip().replace(\" \", \" \")\n", " if s.lower() in _EMPTY:\n", " return _np.nan, None, True\n", " bound = None\n", " if s[:1] in \"<>\":\n", " bound, s = s[0], s[1:].strip()\n", " negative = s.startswith(\"(\") and s.endswith(\")\")\n", " s = _CURRENCY.sub(\"\", s).strip(\"() \").replace(\" \", \"\").replace(\"'\", \"\")\n", " percent = s.endswith(\"%\")\n", " s = s.rstrip(\"%\")\n", " if comma_decimal:\n", " s = s.replace(\".\", \"\").replace(\",\", \".\")\n", " else:\n", " s = s.replace(\",\", \"\")\n", " try:\n", " value = float(s)\n", " except ValueError:\n", " return _np.nan, None, False\n", " if negative:\n", " value = -value\n", " return (value / 100.0 if percent else value), bound, True\n", "\n", "\n", "def _parse_dates(values, column, dayfirst):\n", " text = [str(v).strip() for v in values]\n", " present = [t for t in text if t.lower() not in _EMPTY]\n", " if not present:\n", " return _pd.Series([_pd.NaT] * len(text)), []\n", " notes = []\n", " if all(_re.fullmatch(r\"\\d{8}\", t) for t in present):\n", " return _pd.to_datetime(_pd.Series(text), format=\"%Y%m%d\", errors=\"coerce\"), notes\n", " if all(_re.fullmatch(r\"\\d{10}(\\.\\d+)?|\\d{13}|\\d{16}\", t) for t in present):\n", " digits = len(present[0].split(\".\")[0])\n", " unit = {10: \"s\", 13: \"ms\", 16: \"us\"}[digits]\n", " notes.append(f\"'{column}' holds epoch times in {dict(s='seconds', ms='milliseconds', us='microseconds')[unit]}; read as such.\")\n", " return _pd.to_datetime(_pd.to_numeric(_pd.Series(text), errors=\"coerce\"), unit=unit), notes\n", " cleaned = [_re.sub(r\"\\s+(UTC|GMT|Z)$\", \"\", t) for t in text]\n", " slash = [t for t in cleaned if _re.fullmatch(r\"\\d{1,2}[/.\\-]\\d{1,2}[/.\\-]\\d{2,4}(\\s.*)?\", t)]\n", " if slash and len(slash) >= len(present) * 0.9:\n", " firsts = [int(_re.split(r\"[/.\\-]\", t)[0]) for t in slash]\n", " seconds = [int(_re.split(r\"[/.\\-]\", t)[1]) for t in slash]\n", " dotted = all(\".\" in t.split(\" \")[0] for t in slash)\n", " if dayfirst is None:\n", " if max(firsts) > 12 and max(seconds) <= 12:\n", " dayfirst = True\n", " elif max(seconds) > 12 and max(firsts) <= 12:\n", " dayfirst = False\n", " elif dotted:\n", " dayfirst = True\n", " else:\n", " raise ExportReadError(\n", " f\"The dates in '{column}' (for example {slash[0]}) could be day-first or month-first, \"\n", " \"and nothing in the file says which. Re-run with dayfirst=True for 6 July written \"\n", " \"06/07, or dayfirst=False for June 7 written 06/07 - or export the dates as \"\n", " \"2026-07-06.\")\n", " return _pd.to_datetime(_pd.Series(cleaned), dayfirst=dayfirst, errors=\"coerce\"), notes\n", " parsed = _pd.to_datetime(_pd.Series(cleaned), errors=\"coerce\", format=\"mixed\")\n", " return parsed, notes\n", "\n", "\n", "def read_export(source, columns=None, date_columns=(), id_columns=(), dayfirst=None,\n", " max_unreadable_share=0.02):\n", " \"\"\"Read a downloaded report into a table with this notebook's column names. Returns\n", " (table, notes); raises ExportReadError with a plain-words fix when it cannot read without\n", " guessing.\"\"\"\n", " columns = columns or {}\n", " # Each name's candidates in the caller's order of preference, the name itself first.\n", " wanted = {name: list(dict.fromkeys([_norm(name)] + [_norm(a) for a in aliases]))\n", " for name, aliases in columns.items()}\n", " rows, delimiter = _rows_from(source)\n", " rows = [r for r in rows if any(str(c).strip() for c in r)]\n", " if not rows:\n", " raise ExportReadError(\"The file is empty.\")\n", " notes = []\n", " h = _find_header(rows, wanted)\n", " if h:\n", " notes.append(f\"set aside {h} line(s) above the header (a report title or date range).\")\n", " header = [str(c).strip() for c in rows[h]]\n", " body, totals = [], 0\n", " for r in rows[h + 1:]:\n", " first = next((str(c) for c in r if str(c).strip()), \"\")\n", " if _TOTAL_ROW.match(first):\n", " totals += 1\n", " continue\n", " if [str(c).strip() for c in r] == header:\n", " continue\n", " body.append([str(c) for c in r] + [\"\"] * (len(header) - len(r)))\n", " if totals:\n", " notes.append(f\"set aside {totals} total row(s).\")\n", " frame = _pd.DataFrame([r[:len(header)] for r in body], columns=header)\n", " rename = {}\n", " for name, options in wanted.items():\n", " by_norm = {}\n", " for c in frame.columns:\n", " by_norm.setdefault(_norm(c), c)\n", " match = next((by_norm[o] for o in options if o in by_norm and by_norm[o] not in rename), None)\n", " if match is not None:\n", " rename[match] = name\n", " frame = frame.rename(columns=rename)\n", " missing = [name for name in columns if name not in frame.columns]\n", " if missing:\n", " seen = \", \".join(f\"'{c}'\" for c in header[:12])\n", " raise ExportReadError(\n", " f\"Could not find the column(s) {', '.join(missing)}. The file's columns are {seen}. \"\n", " \"Rename the column in the file, or add its name to the list this cell accepts.\")\n", " dates = set(date_columns)\n", " # The file's own convention, for a column that cannot tell by itself: a clicks column of whole\n", " # numbers says nothing, but its \"1.340\" means one thousand three hundred and forty when the\n", " # file's other columns write decimals with a comma.\n", " every = [str(v) for c in frame.columns if c not in date_columns for v in frame[c].tolist()\n", " if _looks_number(v)]\n", " file_comma = _decimal_comma(every)\n", " if file_comma is None:\n", " file_comma = delimiter == \";\"\n", " ids = set(id_columns) | {c for c in frame.columns if _re.search(r\"(^|[ _])(id|pseudo id)$\", _norm(c))}\n", " for col in frame.columns:\n", " values = frame[col].tolist()\n", " if col in dates:\n", " parsed, extra = _parse_dates(values, col, dayfirst)\n", " notes += extra\n", " bad = int(parsed.isna().sum() - sum(str(v).strip().lower() in _EMPTY for v in values))\n", " if bad > max(1, max_unreadable_share * len(values)):\n", " example = next(v for v, p in zip(values, parsed) if _pd.isna(p) and str(v).strip())\n", " raise ExportReadError(f\"{bad} value(s) in '{col}' are not dates I can read, for example '{example}'.\")\n", " frame[col] = parsed.values\n", " continue\n", " if col in ids:\n", " continue\n", " present = [v for v in values if str(v).strip().lower() not in _EMPTY]\n", " if not present or sum(_looks_number(v) for v in present) < 0.5 * len(present):\n", " continue\n", " comma = _decimal_comma(present)\n", " if comma is None:\n", " comma = file_comma\n", " parsed = [_to_number(v, comma) for v in values]\n", " unreadable = [v for v, (_, _, ok) in zip(values, parsed) if not ok]\n", " if unreadable and col not in columns:\n", " notes.append(f\"'{col}' mixes numbers with labels such as '{unreadable[0]}'; kept as text.\")\n", " continue\n", " if unreadable:\n", " notes.append(f\"{len(unreadable)} value(s) in '{col}' could not be read as numbers, for example \"\n", " f\"'{unreadable[0]}'; they are left empty.\")\n", " if len(unreadable) > max(1, max_unreadable_share * len(values)):\n", " raise ExportReadError(\n", " f\"{len(unreadable)} of {len(values)} values in '{col}' are not numbers I can read, for \"\n", " f\"example '{unreadable[0]}'. Fix them in the file, or export the column unformatted.\")\n", " frame[col] = [p[0] for p in parsed]\n", " bounds = [p[1] for p in parsed]\n", " if any(bounds):\n", " # An object column, so a day with no limit stays None: a newer pandas would infer a string\n", " # column here and turn None into NaN, which is truthy and reads as a limit.\n", " frame[f\"{col}_bound\"] = _pd.Series(bounds, index=frame.index, dtype=object)\n", " notes.append(f\"'{col}' has values the platform shows only as a limit (for example \"\n", " f\"'{next(v for v, b in zip(values, bounds) if b)}'); kept as the limit, with the \"\n", " f\"direction in '{col}_bound'.\")\n", " if dates:\n", " undated = frame[list(dates)].isna().all(axis=1)\n", " if undated.any():\n", " frame = frame[~undated].reset_index(drop=True)\n", " notes.append(f\"set aside {int(undated.sum())} row(s) with no date (GA4 writes its grand total \"\n", " \"this way, under the header).\")\n", " return frame, notes\n", "# ---- end of export reader -------------------------------------------------------------------------" ] }, { "cell_type": "code", "execution_count": 24, "id": "fba9dadc", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T16:55:34.822173Z", "iopub.status.busy": "2026-10-24T16:55:34.821790Z", "iopub.status.idle": "2026-10-24T16:55:34.944755Z", "shell.execute_reply": "2026-10-24T16:55:34.943249Z" } }, "outputs": [], "source": [ "import re\n", "\n", "\n", "class DataProblem(Exception):\n", " \"\"\"Raised with a message written for a person, not a stack trace.\"\"\"\n", "\n", "REQUIRED = (\"date\",)\n", "\n", "def read_daily_csv(path):\n", " \"\"\"Load a daily export and hand back a clean, sorted, typed frame.\"\"\"\n", " # The shared reader sets a download's title lines and Total rows aside, reads \"1,203.40\",\n", " # \"€12.30\", comma decimals and the platforms' date forms, and opens an .xlsx too.\n", " try:\n", " raw, notes = read_export(path, columns={\"date\": [\"date\", \"day\", \"date_\", \"report_date\",\n", " \"reporting starts\"]},\n", " date_columns=[\"date\"])\n", " except ExportReadError as problem:\n", " raise DataProblem(str(problem)) from None\n", " for note in notes:\n", " print(\" \" + note)\n", " # Lower-case the names and drop a currency suffix, so Meta's \"Amount spent (EUR)\" reads as\n", " # \"amount spent\".\n", " raw.columns = [re.sub(r\"\\s*\\((?:[a-z]{3}|[€$£])\\)$\", \"\", str(c).strip().lower())\n", " for c in raw.columns]\n", " # Aliases cover what the platforms actually export. Meta's own column names\n", " # (\"amount spent\", \"results\") are here because the recipe above tells you to\n", " # export exactly those - a loader that then refused them would be the recipe\n", " # and the reader disagreeing.\n", " alias = {\"day\": \"date\", \"date_\": \"date\", \"report_date\": \"date\",\n", " \"spend\": \"cost\", \"cost_eur\": \"cost\", \"amount\": \"cost\",\n", " \"amount spent\": \"cost\", \"amount_spent\": \"cost\",\n", " \"conv\": \"conversions\", \"conversion\": \"conversions\",\n", " \"leads\": \"conversions\", \"results\": \"conversions\",\n", " \"value\": \"metric\", \"cpa\": \"metric\"}\n", " # An alias that collides with a column already present would silently create\n", " # two columns of the same name, and pandas then fails with a raw traceback\n", " # instead of a sentence. Catch it here and say which pair to sort out.\n", " for src_col, dest in alias.items():\n", " if src_col in raw.columns and dest in raw.columns and src_col != dest:\n", " raise DataProblem(\n", " f\"The file has both {src_col!r} and {dest!r}, and I read them as the \"\n", " f\"same thing. Delete or rename one of them so there is a single \"\n", " f\"{dest} column.\")\n", " raw = raw.rename(columns={k: v for k, v in alias.items() if k in raw.columns})\n", " if \"date\" not in raw.columns:\n", " raise DataProblem(\n", " \"I could not find a date column. Rename the column holding the \"\n", " f\"calendar date to 'date'. The file has: {', '.join(raw.columns)}.\")\n", " out = pd.DataFrame({\"date\": pd.to_datetime(raw[\"date\"], errors=\"coerce\",\n", " format=\"mixed\")})\n", " if out[\"date\"].isna().any():\n", " bad = int(out[\"date\"].isna().sum())\n", " raise DataProblem(\n", " f\"{bad} row(s) have a date I cannot read. Use a plain format like \"\n", " \"2026-03-02, and delete any total or subtotal rows at the bottom.\")\n", " if \"metric\" not in raw.columns:\n", " missing = [c for c in (\"cost\", \"conversions\") if c not in raw.columns]\n", " if missing:\n", " raise DataProblem(\n", " \"Without a 'metric' column I need both 'cost' and 'conversions' \"\n", " f\"to work out cost per conversion. Missing: {', '.join(missing)}. \"\n", " \"Either add them, or add a 'metric' column holding the number you \"\n", " \"want analysed.\")\n", " if \"metric\" in raw.columns:\n", " out[\"metric\"] = pd.to_numeric(raw[\"metric\"], errors=\"coerce\")\n", " else:\n", " cost = pd.to_numeric(raw[\"cost\"], errors=\"coerce\")\n", " conv = pd.to_numeric(raw[\"conversions\"], errors=\"coerce\")\n", " out[\"cost\"], out[\"conversions\"] = cost, conv\n", " # A conversions column that never moves is the failure nobody looks for:\n", " # cost/conversions is then just cost rescaled, so the whole analysis is a\n", " # spend analysis wearing a cost-per-conversion label. Both validation\n", " # readers hit this on a test file and neither guard caught it, because the\n", " # zero-variance check downstream only ever inspected the DERIVED metric.\n", " live = conv.dropna()\n", " if len(live) > 2 and live.nunique() == 1:\n", " raise DataProblem(\n", " f\"Every row has exactly {live.iloc[0]:g} conversions. Cost per \"\n", " \"conversion is then just your cost divided by a constant, so this \"\n", " \"would measure spend rather than efficiency. Re-pull the export \"\n", " \"with the real per-day conversion counts.\")\n", " with np.errstate(divide=\"ignore\", invalid=\"ignore\"):\n", " out[\"metric\"] = np.where(conv > 0, cost / conv, np.nan)\n", " out = out.sort_values(\"date\").reset_index(drop=True)\n", " if out[\"date\"].duplicated().any():\n", " n = int(out[\"date\"].duplicated().sum())\n", " raise DataProblem(\n", " f\"{n} date(s) appear more than once. This analysis needs exactly one \"\n", " \"row per day - if your export is split by campaign or device, total \"\n", " \"it up per day first (consolidate_to_daily does this for you).\")\n", " return out\n", "\n", "def consolidate_to_daily(path, date_col=\"date\", cost_col=\"cost\",\n", " conv_col=\"conversions\"):\n", " \"\"\"Total a per-row export (per campaign, per device, per whatever) into one\n", " row per day, which is the shape the analysis needs.\"\"\"\n", " try:\n", " raw, notes = read_export(path, columns={\"date\": [date_col, \"date\", \"day\"]}, date_columns=[\"date\"])\n", " except ExportReadError as problem:\n", " raise DataProblem(str(problem)) from None\n", " for note in notes:\n", " print(\" \" + note)\n", " raw.columns = [re.sub(r\"\\s*\\((?:[a-z]{3}|[€$£])\\)$\", \"\", str(c).strip().lower())\n", " for c in raw.columns]\n", " date_col = \"date\"\n", " for c in (date_col, cost_col, conv_col):\n", " if c not in raw.columns:\n", " raise DataProblem(\n", " f\"Column {c!r} is not in the export. Columns present: \"\n", " f\"{', '.join(raw.columns)}. Pass the right names as arguments, \"\n", " \"e.g. consolidate_to_daily(path, cost_col='spend').\")\n", " raw[date_col] = pd.to_datetime(raw[date_col], errors=\"coerce\")\n", " grouped = (raw.dropna(subset=[date_col])\n", " .groupby(raw[date_col].dt.date, as_index=False)\n", " .agg(cost=(cost_col, \"sum\"), conversions=(conv_col, \"sum\")))\n", " grouped = grouped.rename(columns={grouped.columns[0]: \"date\"})\n", " grouped[\"date\"] = pd.to_datetime(grouped[\"date\"])\n", " return grouped\n" ] }, { "cell_type": "markdown", "id": "eab89df3", "metadata": {}, "source": [ "### Getting the export\n", "\n", "If you do not already have a clean daily file, this is how to get one." ] }, { "cell_type": "code", "execution_count": 25, "id": "e1931b01", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T16:55:34.953802Z", "iopub.status.busy": "2026-10-24T16:55:34.953373Z", "iopub.status.idle": "2026-10-24T16:55:34.971769Z", "shell.execute_reply": "2026-10-24T16:55:34.967978Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "GOOGLE ADS (the two-minute route)\n", " Campaigns > pick the campaign > Reports > Predefined > Time > Day\n", " Columns: Cost, Conversions. Date range: at least 8 weeks BEFORE the change,\n", " through today. Segment by nothing else - one row per day is what you want.\n", " Download > CSV.\n", "\n", "META ADS MANAGER\n", " Ads Reporting > Export > breakdown by Day. Columns: Amount spent, Results.\n", " Same rule: no other breakdown, one row per day.\n", "\n", "GA4 VIA BIGQUERY (when conversions live in GA4 rather than the ad platform)\n", " Run the query below, export the result as CSV, and point the notebook at it.\n", "\n", " SELECT\n", " PARSE_DATE('%Y%m%d', event_date) AS date,\n", " SUM(CASE WHEN event_name = ''\n", " THEN 1 ELSE 0 END) AS conversions,\n", " 0 AS cost\n", " FROM `..events_*`\n", " WHERE _TABLE_SUFFIX BETWEEN '' AND ''\n", " AND collected_traffic_source.manual_source = '' -- the session's source, not the user's first\n", " GROUP BY date\n", " ORDER BY date\n", "\n", " GA4 does not hold your ad spend, so join the cost column from the ad\n", " platform's own daily export before running the analysis.\n", "\n", "WHAT TO WATCH FOR\n", " - One row per day. If the export is split by campaign, device or network,\n", " total it per day first (consolidate_to_daily does this).\n", " - No gaps. A paused day with no rows is a hole, and the weekly part of the\n", " model cannot be formed across one.\n", " - The change date has to be the date the change actually went live, not the\n", " date somebody wrote the ticket.\n", " - Pull at least eight weeks before the change. Three is the floor where the\n", " weekly pattern becomes estimable at all, and eight gives it room.\n", "\n" ] } ], "source": [ "EXPORT_RECIPE = \"\"\"\\\n", "GOOGLE ADS (the two-minute route)\n", " Campaigns > pick the campaign > Reports > Predefined > Time > Day\n", " Columns: Cost, Conversions. Date range: at least 8 weeks BEFORE the change,\n", " through today. Segment by nothing else - one row per day is what you want.\n", " Download > CSV.\n", "\n", "META ADS MANAGER\n", " Ads Reporting > Export > breakdown by Day. Columns: Amount spent, Results.\n", " Same rule: no other breakdown, one row per day.\n", "\n", "GA4 VIA BIGQUERY (when conversions live in GA4 rather than the ad platform)\n", " Run the query below, export the result as CSV, and point the notebook at it.\n", "\n", " SELECT\n", " PARSE_DATE('%Y%m%d', event_date) AS date,\n", " SUM(CASE WHEN event_name = ''\n", " THEN 1 ELSE 0 END) AS conversions,\n", " 0 AS cost\n", " FROM `..events_*`\n", " WHERE _TABLE_SUFFIX BETWEEN '' AND ''\n", " AND collected_traffic_source.manual_source = '' -- the session's source, not the user's first\n", " GROUP BY date\n", " ORDER BY date\n", "\n", " GA4 does not hold your ad spend, so join the cost column from the ad\n", " platform's own daily export before running the analysis.\n", "\n", "WHAT TO WATCH FOR\n", " - One row per day. If the export is split by campaign, device or network,\n", " total it per day first (consolidate_to_daily does this).\n", " - No gaps. A paused day with no rows is a hole, and the weekly part of the\n", " model cannot be formed across one.\n", " - The change date has to be the date the change actually went live, not the\n", " date somebody wrote the ticket.\n", " - Pull at least eight weeks before the change. Three is the floor where the\n", " weekly pattern becomes estimable at all, and eight gives it room.\n", "\"\"\"\n", "\n", "print(EXPORT_RECIPE)" ] }, { "cell_type": "markdown", "id": "b3561da1", "metadata": {}, "source": [ "### The checks that run before any number is produced\n", "\n", "These are not decoration. Each one is a way this analysis can quietly give you a\n", "confident wrong answer, and each one stops it." ] }, { "cell_type": "code", "execution_count": 26, "id": "4bce8b45", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T16:55:35.032075Z", "iopub.status.busy": "2026-10-24T16:55:35.031670Z", "iopub.status.idle": "2026-10-24T16:55:35.213433Z", "shell.execute_reply": "2026-10-24T16:55:35.212519Z" } }, "outputs": [], "source": [ "MIN_PRE_DAYS = 21 # about three weeks - enough to see the weekly shape\n", "MIN_POST_DAYS = 3\n", "FLOOR_PRE_FIT = 14 # below this nothing is fitted at all\n", "\n", "def check_your_data(daily, change_date):\n", " \"\"\"Plain-words problems (stop) and warnings (proceed, but say so).\n", "\n", " These are the honest failure modes, not decoration. The gap check in\n", " particular is the one people are surprised by: the weekly term counts\n", " positions, so across a missing day 'the same weekday last week' is no longer\n", " seven rows back and the seasonal model cannot be formed at all.\n", " \"\"\"\n", " problems, warnings_ = [], []\n", " change = pd.to_datetime(change_date)\n", " d = daily.dropna(subset=[\"metric\"]).copy()\n", " dropped = len(daily) - len(d)\n", " if dropped:\n", " warnings_.append(\n", " f\"{dropped} day(s) had no conversions, so cost per conversion does \"\n", " \"not exist for them. They are dropped - which creates gaps, so read \"\n", " \"the gap note below.\")\n", " pre = d[d.date < change]\n", " post = d[d.date >= change]\n", " if len(pre) < FLOOR_PRE_FIT:\n", " problems.append(\n", " f\"Only {len(pre)} usable day(s) before the change. Nothing can be \"\n", " f\"fitted below {FLOOR_PRE_FIT}. Pull a longer history and re-run.\")\n", " elif len(pre) < MIN_PRE_DAYS:\n", " warnings_.append(\n", " f\"Only {len(pre)} day(s) before the change (about three weeks is the \"\n", " \"point where the weekly pattern becomes estimable). The answer will \"\n", " \"come with a wide range - treat it as a hint, not a finding.\")\n", " if len(post) < MIN_POST_DAYS:\n", " problems.append(\n", " f\"Only {len(post)} day(s) after the change. Wait for at least \"\n", " f\"{MIN_POST_DAYS} and re-run.\")\n", " if len(pre) >= 2:\n", " gaps = pre.date.diff().dt.days.dropna()\n", " n_gaps = int((gaps > 1).sum())\n", " if n_gaps:\n", " warnings_.append(\n", " f\"The days before the change are not consecutive - {n_gaps} \"\n", " \"gap(s). The weekly part of the model counts positions, so it \"\n", " \"cannot be formed across a hole. The simple before/after \"\n", " \"comparison is used instead, and it is the weaker answer.\")\n", " if float(pre.metric.std(ddof=1) or 0) == 0.0:\n", " problems.append(\n", " \"Every day before the change has the same value. There is no \"\n", " \"variation to model - check you exported the metric you meant to.\")\n", " if len(post) >= 2:\n", " post_gaps = int((post.date.diff().dt.days.dropna() > 1).sum())\n", " if post_gaps:\n", " warnings_.append(\n", " f\"The days after the change are not consecutive - {post_gaps} gap(s), usually days \"\n", " \"with no conversions. The projection is lined up by position, so each missing day \"\n", " \"shifts the comparison: read the effect as approximate, or analyse cost per click \"\n", " \"or weekly totals, which have no empty days.\")\n", " return {\"problems\": problems, \"warnings\": warnings_,\n", " \"n_pre\": int(len(pre)), \"n_post\": int(len(post)),\n", " \"contiguous\": bool(len(pre) >= 2 and\n", " int((pre.date.diff().dt.days.dropna() > 1).sum()) == 0)}\n", "\n", "def analyse_your_data(daily, change_date, n_draws=2000):\n", " \"\"\"Run the whole thing on your data, and fall back honestly when it cannot.\"\"\"\n", " verdict = check_your_data(daily, change_date)\n", " if verdict[\"problems\"]:\n", " return {\"ran\": False, **verdict}\n", " d = daily.dropna(subset=[\"metric\"]).copy()\n", " change = pd.to_datetime(change_date)\n", " pre = d[d.date < change].metric.to_numpy()\n", " post = d[d.date >= change].metric.to_numpy()\n", " naive = {\"pre_mean\": float(pre.mean()), \"post_mean\": float(post.mean()),\n", " \"diff\": float(post.mean() - pre.mean())}\n", " if not verdict[\"contiguous\"]:\n", " return {\"ran\": True, \"method\": \"simple before/after comparison\",\n", " \"why\": \"the days before the change are not consecutive\",\n", " \"naive\": naive, **verdict}\n", " res, spec = fit_pre(pre)\n", " diag = diagnostics(res, spec)\n", " if not (diag[\"ljung_box_pass\"] and diag[\"hetero_pass\"]):\n", " return {\"ran\": True, \"method\": \"simple before/after comparison\",\n", " \"why\": \"the model did not pass its own checks on this series\",\n", " \"naive\": naive, \"diagnostics\": diag, **verdict}\n", " horizon, pre_sd = read_horizon(res, len(post), pre)\n", " horizon = max(1, min(horizon, len(post)))\n", " full = mean_effect_ci(res, post[:horizon], n_draws=n_draws)\n", " # Part one calls the placebo sweep \"the check that separates a result from a\n", " # coincidence\", so it runs on YOUR data too. An earlier version computed it\n", " # only for the worked example, which meant the reader was told the check\n", " # mattered and then quietly not given one.\n", " try:\n", " stat = actual_effect_stat(res, post, horizon)\n", " placebo = falsification_sweep(pre, stat, horizon)\n", " except Exception:\n", " placebo = {\"ran\": False, \"reason\": \"sweep could not be formed\"}\n", " return {\"ran\": True, \"method\": \"interrupted time series\", \"naive\": naive,\n", " \"read_window_days\": int(horizon), \"diagnostics\": diag,\n", " \"placebo\": placebo,\n", " \"effect\": full[\"mean_effect\"], \"ci_low\": full[\"ci_low\"],\n", " \"ci_high\": full[\"ci_high\"],\n", " \"excludes_zero\": bool(full[\"ci_high\"] < 0 or full[\"ci_low\"] > 0),\n", " \"draw_failure_share\": full[\"draw_failure_share\"],\n", " **verdict}\n" ] }, { "cell_type": "markdown", "id": "9ebb9f2b", "metadata": {}, "source": [ "### Point it at your file\n", "\n", "Edit the two lines below. That is the only editing you need to do." ] }, { "cell_type": "code", "execution_count": 27, "id": "39503f79", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T16:55:35.243198Z", "iopub.status.busy": "2026-10-24T16:55:35.242607Z", "iopub.status.idle": "2026-10-24T16:55:35.276001Z", "shell.execute_reply": "2026-10-24T16:55:35.275053Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "STOP: Set CHANGE_DATE above to the date your change went live, in YYYY-MM-DD form. It is deliberately not pre-filled: a real-looking default would let you analyse your data against somebody else's change date without noticing.\n" ] } ], "source": [ "from pathlib import Path\n", "\n", "# ---------------------------------------------------------------\n", "YOUR_FILE = \"your_daily_export.csv\" # <- your CSV\n", "CHANGE_DATE = \"YYYY-MM-DD\" # <- the day the change went live\n", "# ---------------------------------------------------------------\n", "# Set this to True ONLY if you want to watch the demonstration account first.\n", "# It analyses invented data, and every line it prints says so.\n", "RUN_THE_DEMO_INSTEAD = False\n", "\n", "def _demo_frame():\n", " demo = generate_series()\n", " d = pd.DataFrame({\"date\": demo.date, \"cost\": (demo.cpa * 11).round(2),\n", " \"conversions\": 11})\n", " d[\"metric\"] = d.cost / d.conversions\n", " return d, demo.date.iloc[N_PRE]\n", "\n", "# This cell REFUSES rather than falling back. An earlier version quietly analysed\n", "# the demonstration account when the filename did not resolve and printed the\n", "# answer in the ordinary format - and those invented numbers happen to match the\n", "# worked example, so anyone checking the output against the article would have\n", "# concluded it looked right. A mistyped filename, an upload landing in the wrong\n", "# folder, or a recycled runtime are all ordinary events, and that path ended with\n", "# someone spending money on fabricated numbers.\n", "stop = None\n", "if RUN_THE_DEMO_INSTEAD:\n", " daily, change_date = _demo_frame()\n", " tag = \"DEMO (invented data) | \"\n", "else:\n", " tag = \"\"\n", " if CHANGE_DATE == \"YYYY-MM-DD\":\n", " stop = (\"Set CHANGE_DATE above to the date your change went live, in \"\n", " \"YYYY-MM-DD form. It is deliberately not pre-filled: a real-looking \"\n", " \"default would let you analyse your data against somebody else's \"\n", " \"change date without noticing.\")\n", " elif not Path(YOUR_FILE).exists():\n", " stop = (f\"No file called {YOUR_FILE!r} in this folder. Check the name and \"\n", " \"that it uploaded. Nothing has been analysed - set \"\n", " \"RUN_THE_DEMO_INSTEAD = True if you want to watch the \"\n", " \"demonstration account first.\")\n", " else:\n", " daily = read_daily_csv(YOUR_FILE)\n", " change_date = CHANGE_DATE\n", "\n", "if stop:\n", " print(\"STOP:\", stop)\n", "else:\n", " result = analyse_your_data(daily, change_date)\n", " for p_ in result[\"problems\"]:\n", " print(tag + \"STOP:\", p_)\n", " for w_ in result[\"warnings\"]:\n", " print(tag + \"NOTE:\", w_)\n", " if result[\"ran\"]:\n", " print()\n", " print(tag + f\"method used: {result['method']}\")\n", " if result[\"method\"] == \"interrupted time series\":\n", " d_ = result[\"diagnostics\"]\n", " print(tag + f\"read window: {result['read_window_days']} days\")\n", " print(tag + f\"effect: {result['effect']:+.2f} per day \"\n", " f\"[{result['ci_low']:+.2f}, {result['ci_high']:+.2f}]\")\n", " print(tag + f\"range excludes zero: {result['excludes_zero']}\")\n", " print()\n", " print(tag + \"the model's own checks on YOUR data:\")\n", " for nm, ok in ((\"residuals independent\", d_[\"ljung_box_pass\"]),\n", " (\"residuals ~normal\", d_[\"jarque_bera_pass\"]),\n", " (\"noise steady\", d_[\"hetero_pass\"])):\n", " print(tag + f\" {nm:24s} {'pass' if ok else 'FAIL'}\")\n", " print()\n", " share = result[\"draw_failure_share\"]\n", " print(tag + f\"interval warrant: {share:.1%} of parameter draws failed to simulate\"\n", " + (\" - past 5% the range loses its warrant; read it as unreliable\"\n", " if share > 0.05 else \" - the range stands\"))\n", " print(tag + \"placebo check: \" + (\n", " \"ran - \" + (\"cleared\" if result[\"placebo\"][\"passes\"] else\n", " \"DID NOT CLEAR, treat this as suggestive only\")\n", " if result.get(\"placebo\", {}).get(\"ran\")\n", " else \"could not run on this series, so nothing here rules out that \"\n", " \"quiet weeks produce swings this large\"))\n", " else:\n", " print(tag + f\"why: {result['why']}\")\n", " print(tag + f\"before/after difference: {result['naive']['diff']:+.2f}\")\n", " print(tag + \"READ THIS AS A HINT, NOT AN ANSWER: the plain before/after comparison has no \"\n", " \"range, and when the metric was already moving before the change it can point \"\n", " \"the wrong way - the reason this notebook exists. On a low-volume lead account \"\n", " \"try cost per click, or weekly totals, which the model can usually fit.\")\n" ] }, { "cell_type": "markdown", "id": "bff13867", "metadata": {}, "source": [ "### Watching it fail properly\n", "\n", "A method you have only seen succeed is one you cannot trust. These are the four\n", "ways your data can defeat this analysis, each one run deliberately." ] }, { "cell_type": "code", "execution_count": 28, "id": "57fdbe8c", "metadata": { "execution": { "iopub.execute_input": "2026-10-24T16:55:35.284439Z", "iopub.status.busy": "2026-10-24T16:55:35.280074Z", "iopub.status.idle": "2026-10-24T16:55:35.407362Z", "shell.execute_reply": "2026-10-24T16:55:35.406551Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--- not enough history before the change\n", " STOP: Only 9 usable day(s) before the change. Nothing can be fitted below 14. Pull a longer history and re-run.\n", "\n", "--- not enough days since the change\n", " STOP: Only 2 day(s) after the change. Wait for at least 3 and re-run.\n", "\n", "--- missing days in the middle\n", " NOTE: The days before the change are not consecutive - 2 gap(s). The weekly part of the model counts positions, so it cannot be formed across a hole. The simple before/after comparison is used instead, and it is the weaker answer.\n", " fell back to: simple before/after comparison\n", "\n", "--- days with no conversions at all\n", " NOTE: 2 day(s) had no conversions, so cost per conversion does not exist for them. They are dropped - which creates gaps, so read the gap note below.\n", " NOTE: The days before the change are not consecutive - 1 gap(s). The weekly part of the model counts positions, so it cannot be formed across a hole. The simple before/after comparison is used instead, and it is the weaker answer.\n", " fell back to: simple before/after comparison\n", "\n" ] } ], "source": [ "def _demo_frame(d):\n", " d = d.copy()\n", " d[\"metric\"] = d.cost / d.conversions.replace(0, np.nan)\n", " return d\n", "\n", "_base = generate_series()\n", "_full = pd.DataFrame({\"date\": _base.date, \"cost\": (_base.cpa * 11).round(2),\n", " \"conversions\": 11})\n", "_change = _base.date.iloc[N_PRE]\n", "\n", "_cases = {\n", " \"not enough history before the change\": _full.iloc[N_PRE - 9:],\n", " \"not enough days since the change\": _full.iloc[:N_PRE + 2],\n", " \"missing days in the middle\": _full.drop(index=[20, 33]),\n", " \"days with no conversions at all\": _full.assign(\n", " conversions=lambda x: x.conversions.mask(x.index.isin([15, 16]), 0)),\n", "}\n", "\n", "for name, frame in _cases.items():\n", " r = analyse_your_data(_demo_frame(frame), _change, n_draws=400)\n", " print(f\"--- {name}\")\n", " for p in r[\"problems\"]:\n", " print(\" STOP:\", p)\n", " for w in r[\"warnings\"]:\n", " print(\" NOTE:\", w)\n", " if r[\"ran\"]:\n", " print(f\" fell back to: {r['method']}\")\n", " print()" ] }, { "cell_type": "markdown", "id": "4bb0946f", "metadata": {}, "source": [ "---\n", "\n", "## What this method claims, and what it does not\n", "\n", "**It claims:** after this specific, dated change, the metric moved away from its\n", "own projected path by this much, with this much uncertainty.\n", "\n", "**It does not claim** that nothing else contributed. If a competitor entered the\n", "auction the same week, or the platform shipped an algorithm update, or demand\n", "shifted, the number blends those in and there is no way to separate them from\n", "inside the data. The placebo test catches breaks the metric produces on its own;\n", "it cannot catch something that happened once, exactly when you made your change.\n", "\n", "The only clean answer to that problem is a controlled test - hold the change back\n", "from a randomly chosen slice of the account and compare. When you can run one,\n", "run one. When you cannot, this is the honest second best, and the range it quotes\n", "is doing real work: it is telling you how much of the answer is evidence and how\n", "much is noise.\n", "\n", "*Synthetic data throughout part one, generated in this notebook under a fixed\n", "seed. Part two runs on your data, locally.*" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 5 }