{ "cells": [ { "cell_type": "markdown", "id": "38c990b5", "metadata": {}, "source": [ "# Something moved - is it a blip, a trend, or a broken thing?\n", "\n", "Conversions are down. Cost per acquisition is up. Somebody screenshots the chart and\n", "asks whether it is a problem.\n", "\n", "There are three different things that sentence can mean, and they are not\n", "interchangeable. A **blip** is one bad day that will be gone tomorrow. A **trend** is a\n", "slow slide that has been running for weeks and is easy to miss because no single day\n", "looks wrong. A **break** is a discrete step to a new level - a tracking tag that stopped\n", "firing, a competitor that entered the auction, a feed that went stale.\n", "\n", "Each one needs a different detector, and each detector is close to blind to the shapes\n", "it was not built for. This notebook builds them, plants all three shapes into synthetic\n", "accounts where the truth is known, and scores every detector against every shape.\n", "\n", "Two results are worth knowing before you read any of the working.\n", "\n", "**A sustained change is far easier to see than a single bad day.** Held to the same\n", "false-alarm cost, a step to a new level is caught half the time once it is about a\n", "seventh the size that a one-day spike needs to be. If you are going to watch one thing,\n", "watch the level, not the day.\n", "\n", "**The alarm threshold is not a statistical constant.** There is no number to look up.\n", "A cut that looks strict on one chart fires constantly across an account of twenty\n", "tracked metrics, so the line has to be read off a budget you choose - how many false alarms a week\n", "your team will really investigate before they start ignoring the alerts.\n", "\n", "**The data in this notebook is synthetic.** It is generated here, with the answer built\n", "in, so every reading can be scored against a truth nobody has to guess at. Run it on\n", "your own numbers in the last section; nothing leaves your machine." ] }, { "cell_type": "code", "execution_count": 1, "id": "86aaeb69", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:18:35.059311Z", "iopub.status.busy": "2026-10-25T13:18:35.057961Z", "iopub.status.idle": "2026-10-25T13:18:36.371328Z", "shell.execute_reply": "2026-10-25T13:18:36.369632Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "seed 20260910, comparison window 28 days\n" ] } ], "source": [ "import json\n", "import warnings\n", "from pathlib import Path\n", "\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import pandas as pd\n", "\n", "SEED = 20260910\n", "rng = np.random.default_rng(SEED)\n", "\n", "PAPER = \"#fdfcf9\"\n", "INK = \"#1f2124\"\n", "MUTED = \"#565c66\"\n", "LINE = \"#e6e2da\"\n", "SIGNAL = \"#0f6e56\"\n", "FLAG = \"#b06a2c\"\n", "NAIVE = \"#2a78d6\"\n", "\n", "plt.rcParams.update({\n", " \"figure.facecolor\": PAPER, \"axes.facecolor\": PAPER, \"savefig.facecolor\": PAPER,\n", " \"axes.edgecolor\": LINE, \"axes.labelcolor\": INK, \"text.color\": INK,\n", " \"xtick.color\": MUTED, \"ytick.color\": MUTED, \"axes.grid\": True,\n", " \"grid.color\": LINE, \"grid.linewidth\": 0.8, \"axes.spines.top\": False,\n", " \"axes.spines.right\": False, \"font.size\": 11, \"figure.dpi\": 140,\n", "})\n", "warnings.filterwarnings(\"ignore\")\n", "\n", "HERE = Path.cwd()\n", "FIGURES = HERE / \"figures\"\n", "DATA = HERE / \"data\"\n", "FIGURES.mkdir(exist_ok=True)\n", "DATA.mkdir(exist_ok=True)\n", "\n", "# Four weeks is the comparison window everywhere below. The reason is not statistical\n", "# folklore: daily marketing data has a weekday rhythm, and four weeks is the shortest\n", "# window in which every weekday appears four times, so a Monday is scored against four\n", "# other Mondays rather than against a weekend.\n", "WINDOW = 28\n", "\n", "print(f\"seed {SEED}, comparison window {WINDOW} days\")" ] }, { "cell_type": "markdown", "id": "eb224dd1", "metadata": {}, "source": [ "## The account I am going to break on purpose\n", "\n", "One synthetic advertiser, one metric per campaign: daily cost per acquisition. The\n", "generator gives every tracked metric three things real accounts have and toy examples usually do\n", "not.\n", "\n", "**A weekday rhythm.** Weekends convert differently from Tuesdays. Any detector that\n", "ignores this flags every Saturday.\n", "\n", "**Autocorrelated noise.** Today looks like yesterday. Marketing days are not\n", "independent draws; a good day tends to be followed by another good day, and that\n", "clustering is what makes eyeballing a chart so unreliable - runs of five bad days\n", "happen constantly with nothing wrong at all.\n", "\n", "**A known truth.** Each tracked metric is generated as quiet, or with a blip, a trend, or a break\n", "planted at a known day. Nothing below reads that truth until it is time to score." ] }, { "cell_type": "code", "execution_count": 2, "id": "4eba6f3a", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:18:36.380913Z", "iopub.status.busy": "2026-10-25T13:18:36.380335Z", "iopub.status.idle": "2026-10-25T13:18:36.410963Z", "shell.execute_reply": "2026-10-25T13:18:36.409565Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " day dow value\n", " 0 0 28.292947\n", " 1 1 27.056793\n", " 2 2 25.462120\n", "\n", "four tracked metrics generated, 200 days each, event planted on day 150\n" ] } ], "source": [ "DOW_EFFECT = np.array([0.00, -0.02, -0.03, -0.02, 0.02, 0.10, 0.12]) # Mon..Sun, multiplicative\n", "BASE_CPA = 24.0\n", "NOISE_SD = 0.09 # lognormal-ish noise on the multiplicative scale\n", "RHO = 0.45 # lag-1 autocorrelation, mid-range for daily marketing series\n", "N_DAYS = 200\n", "EVENT_DAY = 150 # where a planted event starts, when there is one\n", "\n", "\n", "SPIKE_RATE = 0.02 # share of days carrying a one-off data glitch\n", "SPIKE_SIZE = (0.40, 0.90)\n", "\n", "\n", "def make_series(shape, magnitude, n_days=N_DAYS, event_day=EVENT_DAY,\n", " rho=RHO, noise_sd=NOISE_SD, spike_rate=SPIKE_RATE, generator=None):\n", " \"\"\"One campaign-day series with a known planted shape.\n", "\n", " shape: 'quiet' | 'blip' | 'trend' | 'break'\n", " magnitude: size of the planted effect as a fraction of the base level.\n", " blip - the single day is multiplied by (1 + magnitude)\n", " trend - the level slides to (1 + magnitude) over the 21 days after the event\n", " break - the level steps to (1 + magnitude) on the event day and stays\n", "\n", " spike_rate is the share of OTHER days carrying a one-off glitch: a billing\n", " correction, an hour of bot traffic, a tag that double-fired. Real accounts have\n", " these and toy simulations do not, which turns out to decide the whole\n", " robust-versus-textbook comparison further down. Set it to zero for a clean world.\n", " \"\"\"\n", " gen = generator if generator is not None else np.random.default_rng(SEED)\n", " day = np.arange(n_days)\n", " dow = day % 7\n", "\n", " # AR(1) noise on the log scale: today echoes yesterday.\n", " innov = gen.normal(0.0, noise_sd * np.sqrt(1 - rho**2), n_days)\n", " eps = np.empty(n_days)\n", " eps[0] = gen.normal(0.0, noise_sd)\n", " for t in range(1, n_days):\n", " eps[t] = rho * eps[t - 1] + innov[t]\n", "\n", " level = np.ones(n_days)\n", " if spike_rate > 0:\n", " hit = gen.random(n_days) < spike_rate\n", " hit[event_day] = False # never collide with the planted event\n", " size = gen.uniform(*SPIKE_SIZE, n_days) * gen.choice([-1.0, 1.0], n_days)\n", " level = level * np.where(hit, 1.0 + size, 1.0)\n", " if shape == \"blip\":\n", " level[event_day] *= 1.0 + magnitude\n", " elif shape == \"trend\":\n", " ramp = np.clip((day - event_day) / 21.0, 0.0, 1.0)\n", " level *= 1.0 + magnitude * ramp\n", " elif shape == \"break\":\n", " level *= np.where(day >= event_day, 1.0 + magnitude, 1.0)\n", " elif shape != \"quiet\":\n", " raise ValueError(f\"unknown shape {shape!r}\")\n", "\n", " values = BASE_CPA * level * (1 + DOW_EFFECT[dow]) * np.exp(eps)\n", " return pd.DataFrame({\"day\": day, \"dow\": dow, \"value\": values})\n", "\n", "\n", "demo = {\n", " \"quiet\": make_series(\"quiet\", 0.0, generator=np.random.default_rng(1)),\n", " \"blip\": make_series(\"blip\", 1.20, generator=np.random.default_rng(2)),\n", " \"trend\": make_series(\"trend\", 0.30, generator=np.random.default_rng(3)),\n", " \"break\": make_series(\"break\", 0.22, generator=np.random.default_rng(4)),\n", "}\n", "print(demo[\"quiet\"].head(3).to_string(index=False))\n", "print(f\"\\nfour tracked metrics generated, {N_DAYS} days each, event planted on day {EVENT_DAY}\")" ] }, { "cell_type": "markdown", "id": "e81b6d12", "metadata": {}, "source": [ "## Three detectors, written out\n", "\n", "Every one of these is a handful of lines. None of them is machine learning, and that is\n", "the point: an alarm that pauses spend has to be explainable to the person whose spend it\n", "paused.\n", "\n", "### The blip detector: a robust one-day score\n", "\n", "Take the last four weeks, remove the weekday rhythm, remove the local slope, and ask how\n", "far today's leftover sits from the middle of the leftovers. Written as a formula:\n", "\n", "$$ Z_{\\text{robust}} = 0.6745 \\cdot \\frac{x_{\\text{today}} - \\text{median}(r)}{\\text{MAD}(r)} $$\n", "\n", "| Term | What it is |\n", "|---|---|\n", "| $x_{\\text{today}}$ | today's residual, after the weekday rhythm and the local slope are taken out |\n", "| $r$ | the residuals of the *baseline* days - the four weeks before today |\n", "| $\\text{median}(r)$ | the middle residual, the robust stand-in for the average |\n", "| $\\text{MAD}(r)$ | median absolute deviation: the median of $\\lvert r_i - \\text{median}(r) \\rvert$ |\n", "| $0.6745$ | a scaling constant, so the score reads on the same scale as an ordinary z-score |\n", "\n", "**Why the median and the MAD instead of the mean and the standard deviation.** The\n", "spikes you are trying to find are in the history you are measuring against. One extreme\n", "day drags the mean toward itself and inflates the standard deviation, so the next spike\n", "has a lower bar to clear and may not clear it at all. The technical name for how much\n", "contamination an estimate survives is its *breakdown point*, and the arithmetic is\n", "blunt: the mean and the standard deviation have a breakdown point of zero - a single\n", "infinite observation makes the mean infinite - while the median tolerates half the data\n", "going bad. That is the whole argument, and it is measured further down rather than\n", "asserted.\n", "\n", "**Today is left out of its own baseline.** The median, the MAD, the weekday means and\n", "the slope are all fitted on the four weeks *ending yesterday*. Fitting them on a window\n", "that includes today and then scoring today against them is the subtle version of the\n", "same self-masking bug: a genuinely large spike pulls its own baseline toward itself and\n", "scores lower than it should." ] }, { "cell_type": "code", "execution_count": 3, "id": "99e7fd01", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:18:36.420574Z", "iopub.status.busy": "2026-10-25T13:18:36.419978Z", "iopub.status.idle": "2026-10-25T13:18:36.456787Z", "shell.execute_reply": "2026-10-25T13:18:36.454024Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "blip detectors defined\n" ] } ], "source": [ "_PAIR_CACHE = {}\n", "\n", "\n", "def _pair_index(n):\n", " \"\"\"Cached upper-triangle index pair for an n-point window.\"\"\"\n", " if n not in _PAIR_CACHE:\n", " i, j = np.triu_indices(n, k=1)\n", " _PAIR_CACHE[n] = (i, j, (j - i).astype(float))\n", " return _PAIR_CACHE[n]\n", "\n", "\n", "def theil_sen_slope(y):\n", " \"\"\"Median of the pairwise slopes. One outlier cannot tilt it.\"\"\"\n", " i, j, gap = _pair_index(len(y))\n", " return float(np.median((y[j] - y[i]) / gap))\n", "\n", "\n", "def dow_means(values, dow):\n", " \"\"\"The average level of each weekday in a window.\"\"\"\n", " m = np.zeros(7)\n", " for d in range(7):\n", " mask = dow == d\n", " if mask.sum() > 0:\n", " m[d] = values[mask].mean()\n", " return m\n", "\n", "\n", "def robust_scores(values, dow, window=WINDOW):\n", " \"\"\"Modified z-score for every day, fitted on the window ENDING YESTERDAY.\"\"\"\n", " values = np.asarray(values, dtype=float)\n", " dow = np.asarray(dow, dtype=int)\n", " n = len(values)\n", " out = np.full(n, np.nan)\n", " x = np.arange(window, dtype=float)\n", " for t in range(window, n):\n", " base_v = values[t - window:t]\n", " base_d = dow[t - window:t]\n", " means = dow_means(base_v, base_d)\n", " adj = base_v - means[base_d]\n", " slope = theil_sen_slope(adj)\n", " resid = adj - slope * x\n", " med = np.median(resid)\n", " mad = np.median(np.abs(resid - med))\n", " if mad <= 0:\n", " continue\n", " # Place today on the same footing: its own weekday mean, the same fitted slope.\n", " today_r = values[t] - means[dow[t]] - slope * window\n", " out[t] = 0.6745 * (today_r - med) / mad\n", " return out\n", "\n", "\n", "def naive_scores(values, dow, window=WINDOW):\n", " \"\"\"The textbook control chart: mean and standard deviation, same window.\"\"\"\n", " n = len(values)\n", " out = np.full(n, np.nan)\n", " for t in range(window, n):\n", " base = values[t - window:t].astype(float)\n", " sd = base.std(ddof=1)\n", " if sd <= 0:\n", " continue\n", " out[t] = (values[t] - base.mean()) / sd\n", " return out\n", "\n", "\n", "print(\"blip detectors defined\")" ] }, { "cell_type": "markdown", "id": "04f1b03c", "metadata": {}, "source": [ "### The trend detector: a slope, scored against the slopes you usually see\n", "\n", "A slow slide never produces a bad *day*. Every day sits comfortably inside the normal\n", "range; it is the direction that is wrong. So the quantity to score is not today's value\n", "but the slope of the last four weeks:\n", "\n", "$$ \\text{slope} = \\text{median}\\left\\{ \\frac{y_j - y_i}{j - i} \\ : \\ i < j \\right\\} $$\n", "\n", "This is the Theil-Sen estimator - the median of every pairwise slope in the window. One\n", "freak day changes one term in that median and nothing else, where a least-squares line\n", "would tilt to chase it.\n", "\n", "Then the same trick as before: a slope on its own means nothing until you know what\n", "slopes this metric usually produces. Score today's four-week slope against the\n", "distribution of four-week slopes over the preceding stretch, using the same median-and-MAD\n", "form.\n", "\n", "**Why a magnitude and not a p-value.** The obvious move is a significance test on the\n", "trend. It does not survive contact with daily marketing data, because those tests assume\n", "the days are independent draws and they are not. Hirsch and Slack, studying exactly this\n", "problem for seasonal water-quality series, found that with serially dependent data the\n", "seasonal trend test's true error rate runs well above its nominal level even at lag-one\n", "correlation as low as 0.2, and that their own correlation-robust modification only holds\n", "up when the autoregressive parameter stays below 0.6 *and* the record is long. Daily\n", "marketing series routinely sit in that range with a record of weeks. A p-value computed\n", "there is a number with a decimal point and no guarantee behind it, so this notebook\n", "scores magnitude instead and prices the false alarms directly." ] }, { "cell_type": "code", "execution_count": 4, "id": "27d520d5", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:18:36.484202Z", "iopub.status.busy": "2026-10-25T13:18:36.483202Z", "iopub.status.idle": "2026-10-25T13:18:36.497351Z", "shell.execute_reply": "2026-10-25T13:18:36.495616Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "trend detector defined\n" ] } ], "source": [ "def rolling_slopes(values, dow, window=WINDOW):\n", " \"\"\"The four-week Theil-Sen slope ending on each day, on de-seasonalised values.\"\"\"\n", " values = np.asarray(values, dtype=float)\n", " dow = np.asarray(dow, dtype=int)\n", " n = len(values)\n", " out = np.full(n, np.nan)\n", " for t in range(window, n + 1):\n", " base_v = values[t - window:t]\n", " base_d = dow[t - window:t]\n", " means = dow_means(base_v, base_d)\n", " out[t - 1] = theil_sen_slope(base_v - means[base_d])\n", " return out\n", "\n", "\n", "def trend_scores(values, dow, window=WINDOW, history=60):\n", " \"\"\"Score today's slope against the slopes this metric usually produces.\"\"\"\n", " slopes = rolling_slopes(values, dow, window)\n", " n = len(values)\n", " out = np.full(n, np.nan)\n", " for t in range(window + history, n):\n", " past = slopes[t - history:t]\n", " past = past[~np.isnan(past)]\n", " if len(past) < history // 2 or np.isnan(slopes[t]):\n", " continue\n", " med = np.median(past)\n", " mad = np.median(np.abs(past - med))\n", " if mad <= 0:\n", " continue\n", " out[t] = 0.6745 * (slopes[t] - med) / mad\n", " return out\n", "\n", "\n", "print(\"trend detector defined\")" ] }, { "cell_type": "markdown", "id": "89a35aed", "metadata": {}, "source": [ "### The break detector: find the best place to cut the series in two\n", "\n", "A step change is a different question again. It is not about one day and not about a\n", "direction - it is about whether the series is better described as one level or as two\n", "levels with a boundary between them.\n", "\n", "That is a search problem with an exact answer. For a candidate segmentation, add up how\n", "badly each segment is described by its own mean, then charge a penalty per boundary so\n", "the search cannot cheat by cutting everywhere:\n", "\n", "$$ \\min_{\\tau_1 \\dots \\tau_k} \\ \\sum_{i=0}^{k} C\\!\\left(y_{\\tau_i+1:\\tau_{i+1}}\\right) + \\beta k $$\n", "\n", "| Term | What it is |\n", "|---|---|\n", "| $\\tau_i$ | the $i$-th boundary - the day the level changed |\n", "| $C(\\cdot)$ | the cost of describing one segment with a single level; here, its sum of squared deviations |\n", "| $\\beta$ | the penalty charged per boundary; higher means only large breaks are worth cutting for |\n", "| $k$ | how many boundaries the segmentation uses |\n", "\n", "Written naively this is an exponential search. The trick that makes it linear in\n", "practice is a pruning rule: once a candidate boundary is provably worse than the best\n", "path so far, it can never become optimal later, so it is dropped from the search\n", "permanently. That is PELT - pruned exact linear time - and it is exact, not approximate.\n", "The implementation below is the whole algorithm in about twenty lines.\n", "\n", "One honest limitation, stated by the algorithm's own authors: the cost function treats\n", "the days *inside* a segment as independent draws. Marketing days are not - that is the\n", "same autocorrelation as above - and the effect is that stable series get cut more often\n", "than they should. The paper says the extension to dependence \"is trivial\" but does not\n", "carry it out, and this notebook does not either. What it does instead is measure the\n", "false-cut rate on quiet tracked metrics, which is where that assumption shows up." ] }, { "cell_type": "code", "execution_count": 5, "id": "c4232a5f", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:18:36.507834Z", "iopub.status.busy": "2026-10-25T13:18:36.507468Z", "iopub.status.idle": "2026-10-25T13:18:36.545876Z", "shell.execute_reply": "2026-10-25T13:18:36.543597Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "break detector defined\n" ] } ], "source": [ "def segment_cost(csum, csum2, a, b):\n", " \"\"\"Sum of squared deviations of y[a:b] about its own mean, from prefix sums.\"\"\"\n", " n = b - a\n", " if n <= 0:\n", " return 0.0\n", " s = csum[b] - csum[a]\n", " s2 = csum2[b] - csum2[a]\n", " return float(s2 - s * s / n)\n", "\n", "\n", "def pelt(values, penalty, min_size=7):\n", " \"\"\"Exact optimal partitioning with PELT pruning. Returns boundary indices.\"\"\"\n", " y = np.asarray(values, dtype=float)\n", " n = len(y)\n", " csum = np.concatenate([[0.0], np.cumsum(y)])\n", " csum2 = np.concatenate([[0.0], np.cumsum(y * y)])\n", "\n", " best = np.full(n + 1, np.inf)\n", " best[0] = -penalty\n", " prev = np.zeros(n + 1, dtype=int)\n", " candidates = [0]\n", "\n", " for t in range(min_size, n + 1):\n", " costs, usable = [], []\n", " for s in candidates:\n", " # A candidate closer than min_size cannot be used YET. It is not out of\n", " # the running - it becomes usable as t grows - so it must be carried\n", " # forward rather than pruned. (Pruning it here is the bug that makes a\n", " # segmentation return no cuts at all: every new candidate is dropped on\n", " # the very next step, and only the no-cut path ever survives.)\n", " if t - s < min_size:\n", " costs.append(np.inf)\n", " usable.append(False)\n", " continue\n", " costs.append(best[s] + segment_cost(csum, csum2, s, t) + penalty)\n", " usable.append(True)\n", " k = int(np.argmin(costs))\n", " best[t] = costs[k]\n", " prev[t] = candidates[k]\n", " # The pruning rule proper: a candidate whose best path to t is already worse\n", " # than the best path to t can never become optimal for any later t.\n", " candidates = [s for s, c, ok in zip(candidates, costs, usable)\n", " if (not ok) or (c - penalty <= best[t])] + [t]\n", "\n", " cuts, t = [], n\n", " while t > 0:\n", " s = prev[t]\n", " if s > 0:\n", " cuts.append(s)\n", " t = s\n", " return sorted(cuts)\n", "\n", "\n", "def break_days(values, dow, lookback=90, penalty_sd=3.0):\n", " \"\"\"Run the segmentation on the recent stretch; return detected boundary days.\"\"\"\n", " n = len(values)\n", " if n < lookback:\n", " return []\n", " v = values[-lookback:].astype(float).copy()\n", " d = dow[-lookback:]\n", " for dd in range(7):\n", " m = d == dd\n", " if m.sum() > 0:\n", " v[m] -= v[m].mean()\n", " # Scale the penalty to the series' own robust noise level, so one setting works\n", " # across metrics measured in euros, per-cent and counts alike.\n", " diffs = np.diff(v)\n", " scale = np.median(np.abs(diffs - np.median(diffs))) / 0.6745 / np.sqrt(2)\n", " scale = max(scale, 1e-9)\n", " beta = penalty_sd * (scale**2) * np.log(lookback)\n", " return [n - lookback + c for c in pelt(v, beta)]\n", "\n", "\n", "print(\"break detector defined\")" ] }, { "cell_type": "markdown", "id": "5f09eac1", "metadata": {}, "source": [ "## Setting the alarm line, before comparing anything\n", "\n", "Here is where most write-ups quietly cheat. They pick a threshold that looks\n", "authoritative - three standard deviations, say - run their favourite detector against a\n", "rival, and report that theirs caught more. Of course it did: a detector set to fire more\n", "easily catches more of everything, including nothing.\n", "\n", "The only fair comparison holds the **false-alarm rate** equal and then asks what each\n", "detector catches. So before any scoring, every detector is calibrated on tracked metrics where\n", "nothing was planted, until they all cost the same.\n", "\n", "The budget below is one false alarm per tracked metric per year. That is not a statistical\n", "constant - there isn't one - it is an operating choice, and further down you can see\n", "what other choices cost." ] }, { "cell_type": "code", "execution_count": 6, "id": "e4656587", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:18:36.555447Z", "iopub.status.busy": "2026-10-25T13:18:36.554660Z", "iopub.status.idle": "2026-10-25T13:19:16.373551Z", "shell.execute_reply": "2026-10-25T13:19:16.369854Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "calibrated on 43,000 quiet metric-days, budget 1 false alarm per tracked metric per year\n", "\n", " robust one-day score fires above 12.30\n", " textbook one-day score fires above 7.98\n", " trend score fires above 4.40\n", " segmentation penalty 9.0 (0.216 false cuts per quiet run, allowance 0.247)\n" ] } ], "source": [ "FALSE_ALARM_BUDGET = 1.0 / 365.0 # per tracked metric, per day\n", "CALIB_REPS = 250\n", "FOLLOW = 28 # how long a detector gets to notice\n", "EVENT_WINDOW = 14 # a cut this close to the event counts as catching it\n", "\n", "\n", "def quiet_panel(reps=CALIB_REPS):\n", " \"\"\"Every detector's raw output on tracked metrics where nothing happened.\"\"\"\n", " robust, naive, trend, series = [], [], [], []\n", " for rep in range(reps):\n", " g = np.random.default_rng(41_000 + rep)\n", " df = make_series(\"quiet\", 0.0, generator=g)\n", " v, d = df[\"value\"].to_numpy(), df[\"dow\"].to_numpy()\n", " robust.append(robust_scores(v, d))\n", " naive.append(naive_scores(v, d))\n", " trend.append(trend_scores(v, d))\n", " series.append((v, d))\n", " return robust, naive, trend, series\n", "\n", "\n", "Q_ROBUST, Q_NAIVE, Q_TREND, Q_SERIES = quiet_panel()\n", "\n", "\n", "def calibrate_score_cut(pools, budget=FALSE_ALARM_BUDGET):\n", " \"\"\"The score above which a quiet tracked metric fires exactly `budget` of the time.\"\"\"\n", " flat = np.concatenate([np.abs(p[~np.isnan(p)]) for p in pools])\n", " return float(np.quantile(flat, 1.0 - budget)), len(flat)\n", "\n", "\n", "def calibrate_penalty(series, budget=FALSE_ALARM_BUDGET, lookback=90):\n", " \"\"\"The lightest penalty whose false cuts on quiet metrics stay inside the budget.\n", "\n", " Coarse pass then a fine pass around it - a single fine sweep would run the\n", " segmentation tens of thousands of times for precision the penalty does not need.\n", " \"\"\"\n", " allowed = budget * lookback # expected false cuts per quiet run\n", "\n", " def rate_at(penalty_sd):\n", " return float(np.mean([len(break_days(v, d, lookback=lookback,\n", " penalty_sd=penalty_sd))\n", " for v, d in series]))\n", "\n", " coarse = None\n", " for penalty_sd in np.arange(2.0, 41.0, 2.0):\n", " if rate_at(penalty_sd) <= allowed:\n", " coarse = float(penalty_sd)\n", " break\n", " if coarse is None:\n", " return 40.0, rate_at(40.0), allowed\n", " best, best_rate = coarse, rate_at(coarse)\n", " for penalty_sd in np.arange(coarse - 1.5, coarse, 0.5):\n", " r = rate_at(float(penalty_sd))\n", " if r <= allowed:\n", " best, best_rate = float(penalty_sd), r\n", " break\n", " return best, best_rate, allowed\n", "\n", "\n", "CUT_ROBUST, n_robust = calibrate_score_cut(Q_ROBUST)\n", "CUT_NAIVE, _ = calibrate_score_cut(Q_NAIVE)\n", "CUT_TREND, _ = calibrate_score_cut(Q_TREND)\n", "PENALTY, quiet_cut_rate, cut_allowance = calibrate_penalty(Q_SERIES)\n", "\n", "print(f\"calibrated on {n_robust:,} quiet metric-days, budget \"\n", " f\"{FALSE_ALARM_BUDGET * 365:.0f} false alarm per tracked metric per year\\n\")\n", "print(f\" robust one-day score fires above {CUT_ROBUST:.2f}\")\n", "print(f\" textbook one-day score fires above {CUT_NAIVE:.2f}\")\n", "print(f\" trend score fires above {CUT_TREND:.2f}\")\n", "print(f\" segmentation penalty {PENALTY:.1f} \"\n", " f\"({quiet_cut_rate:.3f} false cuts per quiet run, allowance {cut_allowance:.3f})\")" ] }, { "cell_type": "markdown", "id": "384b19a6", "metadata": {}, "source": [ "Two things are already visible in those four numbers, before a single event is planted.\n", "\n", "**The one-day scores do not fire anywhere near where a textbook would put them.** A\n", "score built from four weeks of history is not a z-score from a normal distribution, and\n", "treating it as one is wrong by a wide margin. The spread estimate in the denominator is\n", "itself computed from twenty-eight numbers, so it wobbles; a ratio whose denominator\n", "wobbles has far heavier tails than the bell curve the \"three sigma\" reflex assumes. The\n", "only way to know where the line goes is to measure it on your own quiet data, which is\n", "what the cell above does.\n", "\n", "**The segmentation penalty has to be far heavier than the textbook formula suggests.**\n", "That is the autocorrelation problem showing up in the open. The cost function treats the\n", "days inside a segment as independent draws; they are not, so a stable-but-drifting\n", "series looks like several short regimes and gets chopped into them. Killick, Fearnhead\n", "and Eckley say as much in the paper that introduced the algorithm - the data within a\n", "segment are assumed independent \"although extensions to dependence within a segment is\n", "trivial\" - and then do not carry that extension out. Neither does this notebook. What it\n", "does instead is pay for the assumption in the penalty, and show you the bill." ] }, { "cell_type": "markdown", "id": "024df01d", "metadata": {}, "source": [ "## What each detector sees on each shape\n", "\n", "Four tracked metrics - one quiet, one with a blip, one with a trend, one with a break - and the\n", "detectors run over all four at the lines just calibrated." ] }, { "cell_type": "code", "execution_count": 7, "id": "e156b7f8", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:19:16.384932Z", "iopub.status.busy": "2026-10-25T13:19:16.384488Z", "iopub.status.idle": "2026-10-25T13:19:17.924321Z", "shell.execute_reply": "2026-10-25T13:19:17.921479Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "wrote fig_shapes.png\n" ] } ], "source": [ "fig, axes = plt.subplots(4, 1, figsize=(9.5, 10.5), sharex=True)\n", "titles = {\n", " \"quiet\": \"Quiet - nothing planted\",\n", " \"blip\": \"Blip - one bad day\",\n", " \"trend\": \"Trend - a slow slide starting day 150\",\n", " \"break\": \"Break - a step to a new level on day 150\",\n", "}\n", "for ax, (name, df) in zip(axes, demo.items()):\n", " v, d = df[\"value\"].to_numpy(), df[\"dow\"].to_numpy()\n", " ax.plot(df[\"day\"], v, color=MUTED, lw=1.0)\n", " rz = np.abs(np.nan_to_num(robust_scores(v, d), nan=0.0))\n", " tz = np.abs(np.nan_to_num(trend_scores(v, d), nan=0.0))\n", " bd = break_days(v, d, penalty_sd=PENALTY)\n", " for x in np.where(rz > CUT_ROBUST)[0]:\n", " ax.axvline(x, color=FLAG, lw=1.3, alpha=0.8)\n", " for x in np.where(tz > CUT_TREND)[0][:1]:\n", " ax.axvline(x, color=SIGNAL, lw=1.7, ls=\"--\", alpha=0.9)\n", " for x in bd:\n", " ax.axvline(x, color=NAIVE, lw=1.7, ls=\":\", alpha=0.9)\n", " if name != \"quiet\":\n", " ax.axvspan(EVENT_DAY - 0.5, EVENT_DAY + 0.5, color=INK, alpha=0.10)\n", " ax.set_title(titles[name], loc=\"left\", fontsize=11, color=INK)\n", " ax.set_ylabel(\"CPA (EUR)\")\n", "axes[-1].set_xlabel(\"day\")\n", "handles = [\n", " plt.Line2D([], [], color=FLAG, lw=1.3, label=\"blip detector fired\"),\n", " plt.Line2D([], [], color=SIGNAL, lw=1.7, ls=\"--\", label=\"trend detector fired\"),\n", " plt.Line2D([], [], color=NAIVE, lw=1.7, ls=\":\", label=\"break detector cut here\"),\n", "]\n", "axes[0].legend(handles=handles, loc=\"upper left\", frameon=False, fontsize=9)\n", "fig.suptitle(\"One account, four tracked metrics, three detectors (synthetic data)\",\n", " fontsize=12.5, color=INK, x=0.02, ha=\"left\")\n", "fig.tight_layout()\n", "fig.savefig(FIGURES / \"fig_shapes.png\", bbox_inches=\"tight\", dpi=150)\n", "plt.close(fig)\n", "print(\"wrote fig_shapes.png\")" ] }, { "cell_type": "markdown", "id": "9872c17a", "metadata": {}, "source": [ "## The robustness argument, tested rather than repeated\n", "\n", "Every textbook makes the same case for the median and the MAD, and this notebook made it\n", "too, a few cells up. One extreme day drags the mean toward itself and inflates the\n", "standard deviation, so the next spike has a lower bar to clear. The estimator's\n", "*breakdown point* - the share of the data that can go bad before the estimate does -\n", "is 0.5 for the median and zero for the mean.\n", "\n", "That argument is correct and it is not the whole story, because it silently assumes both\n", "detectors are held at the **same threshold**. They should not be. A robust score built\n", "from twenty-eight numbers has much heavier tails than a bell curve, so at any shared\n", "threshold it fires far more often - and it will of course catch more, at a cost the\n", "comparison never shows.\n", "\n", "So test it properly: calibrate each detector to the same false-alarm rate, in two\n", "different worlds.\n", "\n", "**A clean world**, where the only thing in the history is ordinary noise.\n", "\n", "**A realistic world**, where about one day in fifty carries a glitch - a billing\n", "correction, an hour of bot traffic, a tag that double-fired. Every account has these.\n", "\n", "The robustness argument predicts nothing about the clean world and everything about the\n", "dirty one. That is exactly what makes it testable." ] }, { "cell_type": "code", "execution_count": 8, "id": "7ef46e3b", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:19:17.929885Z", "iopub.status.busy": "2026-10-25T13:19:17.929496Z", "iopub.status.idle": "2026-10-25T13:20:30.562725Z", "shell.execute_reply": "2026-10-25T13:20:30.559336Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " world shape cut_robust cut_textbook robust textbook gap_se\n", " clean history blip 6.01 3.51 0.9950 1.0000 0.003531\n", "realistic history blip 13.15 7.95 0.3975 0.3425 0.027051\n" ] } ], "source": [ "WORLDS = {\"clean history\": 0.0, \"realistic history\": SPIKE_RATE}\n", "WORLD_REPS = 400\n", "WORLD_BLIP = 0.95 # the half-catch size measured below: neither detector catches every one\n", "\n", "\n", "def world_panel(spike_rate, reps=CALIB_REPS):\n", " \"\"\"Quiet-metric scores for both one-day detectors in one world.\"\"\"\n", " robust, naive = [], []\n", " for rep in range(reps):\n", " g = np.random.default_rng(61_000 + rep)\n", " df = make_series(\"quiet\", 0.0, spike_rate=spike_rate, generator=g)\n", " v, d = df[\"value\"].to_numpy(), df[\"dow\"].to_numpy()\n", " robust.append(robust_scores(v, d))\n", " naive.append(naive_scores(v, d))\n", " return robust, naive\n", "\n", "\n", "world_rows = []\n", "for world, spike_rate in WORLDS.items():\n", " q_rob, q_nai = world_panel(spike_rate)\n", " cut_rob, _ = calibrate_score_cut(q_rob)\n", " cut_nai, _ = calibrate_score_cut(q_nai)\n", " # One bad day, at the size the one-day score catches about half the time. A bigger day is\n", " # caught by both every time, and a step is missed by both (the grid shows the one-day\n", " # scores are blind to steps), so neither can tell the estimators apart. The event sits\n", " # above most glitches (40 to 90 per cent), so the question under test is \"does the\n", " # estimator survive a dirty history\", not \"can a glitch be told from an identically sized\n", " # glitch\" - which is a question about causes, not statistics.\n", " for shape, mag in [(\"blip\", WORLD_BLIP)]:\n", " caught_r = caught_n = 0\n", " diffs = []\n", " for rep in range(WORLD_REPS):\n", " g = np.random.default_rng(62_000 + rep)\n", " df = make_series(shape, mag, spike_rate=spike_rate, generator=g)\n", " v, d = df[\"value\"].to_numpy(), df[\"dow\"].to_numpy()\n", " rz = np.abs(np.nan_to_num(robust_scores(v, d), nan=0.0))\n", " nz = np.abs(np.nan_to_num(naive_scores(v, d), nan=0.0))\n", " hit_r = int((rz[EVENT_DAY:EVENT_DAY + FOLLOW] > cut_rob).any())\n", " hit_n = int((nz[EVENT_DAY:EVENT_DAY + FOLLOW] > cut_nai).any())\n", " caught_r += hit_r\n", " caught_n += hit_n\n", " diffs.append(hit_r - hit_n)\n", " world_rows.append({\n", " \"world\": world, \"shape\": shape,\n", " \"cut_robust\": round(cut_rob, 2), \"cut_textbook\": round(cut_nai, 2),\n", " \"robust\": caught_r / WORLD_REPS, \"textbook\": caught_n / WORLD_REPS,\n", " \"gap_se\": float(np.std(diffs, ddof=1) / np.sqrt(WORLD_REPS)),\n", " })\n", "worlds = pd.DataFrame(world_rows)\n", "print(worlds.to_string(index=False))" ] }, { "cell_type": "code", "execution_count": 9, "id": "fe3acfec", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:20:30.570775Z", "iopub.status.busy": "2026-10-25T13:20:30.570215Z", "iopub.status.idle": "2026-10-25T13:20:30.889716Z", "shell.execute_reply": "2026-10-25T13:20:30.887568Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "wrote fig_robustness.png\n", "\n", "robust minus textbook, one bad day at +95%, 400 metrics per world:\n", " clean history -0.5 points (standard error 0.4)\n", " realistic history +5.5 points (standard error 2.7)\n" ] } ], "source": [ "fig, ax = plt.subplots(figsize=(8.8, 4.4))\n", "labels = [f\"{r['shape']}\\n{r['world']}\" for _, r in worlds.iterrows()]\n", "x = np.arange(len(worlds))\n", "ax.bar(x - 0.19, worlds[\"textbook\"] * 100, width=0.36, color=NAIVE,\n", " label=\"textbook (mean and standard deviation)\")\n", "ax.bar(x + 0.19, worlds[\"robust\"] * 100, width=0.36, color=SIGNAL,\n", " label=\"robust (median and MAD)\")\n", "ax.set_xticks(x, labels, fontsize=9)\n", "ax.set_ylabel(\"caught, %\")\n", "ax.set_ylim(0, 100)\n", "ax.set_title(\"Both cost the same in false alarms. Only the dirty history separates them.\",\n", " loc=\"left\", fontsize=12, color=INK)\n", "ax.legend(frameon=False, fontsize=9, loc=\"upper right\")\n", "fig.tight_layout()\n", "fig.savefig(FIGURES / \"fig_robustness.png\", bbox_inches=\"tight\", dpi=150)\n", "plt.close(fig)\n", "print(\"wrote fig_robustness.png\")\n", "\n", "clean_row = worlds[worlds[\"world\"] == \"clean history\"].iloc[0]\n", "dirty_row = worlds[worlds[\"world\"] == \"realistic history\"].iloc[0]\n", "clean_gap, clean_se = float(clean_row[\"robust\"] - clean_row[\"textbook\"]), float(clean_row[\"gap_se\"])\n", "dirty_gap, dirty_se = float(dirty_row[\"robust\"] - dirty_row[\"textbook\"]), float(dirty_row[\"gap_se\"])\n", "print(f\"\\nrobust minus textbook, one bad day at +{WORLD_BLIP:.0%}, {WORLD_REPS} metrics per world:\")\n", "print(f\" clean history {clean_gap * 100:+.1f} points (standard error {clean_se * 100:.1f})\")\n", "print(f\" realistic history {dirty_gap * 100:+.1f} points (standard error {dirty_se * 100:.1f})\")" ] }, { "cell_type": "markdown", "id": "a7e56a2e", "metadata": {}, "source": [ "## Scoring every detector against every shape\n", "\n", "Now the experiment. Many independent tracked metrics of each shape, every detector run over all of\n", "them, at the lines calibrated above so each one costs the same in false alarms. A\n", "detector \"catches\" an event if it fires within four weeks of it - and because four weeks\n", "is a long time to wait, the table also records how long it took." ] }, { "cell_type": "code", "execution_count": 10, "id": "7ccc9bee", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:20:30.897560Z", "iopub.status.busy": "2026-10-25T13:20:30.897023Z", "iopub.status.idle": "2026-10-25T13:21:43.371917Z", "shell.execute_reply": "2026-10-25T13:21:43.368987Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " textbook one-day robust one-day trend segmentation\n", "shape \n", "blip 57.8 71.7 6.1 20.0\n", "trend 5.6 10.0 28.9 97.8\n", "break 4.4 6.1 32.2 80.6\n", "quiet 9.4 9.4 3.3 5.6\n", "\n", "median days to notice (blank = never noticed):\n", " textbook one-day lag robust one-day lag trend lag segmentation lag\n", "shape \n", "blip 0.0 0.0 8.0 2.5\n", "trend 13.5 12.0 17.0 11.0\n", "break 3.0 3.0 7.5 0.0\n" ] } ], "source": [ "N_REPS = 180\n", "SHAPES = {\"quiet\": 0.0, \"blip\": 1.20, \"trend\": 0.30, \"break\": 0.22}\n", "SHAPE_SEED = {\"quiet\": 100_000, \"blip\": 200_000, \"trend\": 300_000, \"break\": 400_000}\n", "DETECTORS = [\"textbook one-day\", \"robust one-day\", \"trend\", \"segmentation\"]\n", "\n", "\n", "def first_fire(flags, lo, hi):\n", " \"\"\"Days from lo until the first fire in [lo, hi), or None.\"\"\"\n", " hits = np.where(flags[lo:hi])[0]\n", " return int(hits[0]) if len(hits) else None\n", "\n", "\n", "def detector_flags(v, d):\n", " \"\"\"Per-day fire flags for all four detectors on one tracked metric.\"\"\"\n", " nz = np.abs(np.nan_to_num(naive_scores(v, d), nan=0.0)) > CUT_NAIVE\n", " rz = np.abs(np.nan_to_num(robust_scores(v, d), nan=0.0)) > CUT_ROBUST\n", " tz = np.abs(np.nan_to_num(trend_scores(v, d), nan=0.0)) > CUT_TREND\n", " seg = np.zeros(len(v), dtype=bool)\n", " for c in break_days(v, d, penalty_sd=PENALTY):\n", " if 0 <= c < len(v):\n", " seg[c] = True\n", " return {\"textbook one-day\": nz, \"robust one-day\": rz, \"trend\": tz, \"segmentation\": seg}\n", "\n", "\n", "grid_rows = []\n", "for shape, mag in SHAPES.items():\n", " for rep in range(N_REPS):\n", " g = np.random.default_rng(SHAPE_SEED[shape] + rep)\n", " df = make_series(shape, mag, generator=g)\n", " v, d = df[\"value\"].to_numpy(), df[\"dow\"].to_numpy()\n", " flags = detector_flags(v, d)\n", " row = {\"shape\": shape}\n", " for name, f in flags.items():\n", " lag = first_fire(f, EVENT_DAY, EVENT_DAY + FOLLOW)\n", " row[name] = lag is not None\n", " row[name + \" lag\"] = lag\n", " grid_rows.append(row)\n", "grid = pd.DataFrame(grid_rows)\n", "\n", "catch = grid.groupby(\"shape\")[DETECTORS].mean().reindex([\"blip\", \"trend\", \"break\", \"quiet\"])\n", "lags = grid.groupby(\"shape\")[[c + \" lag\" for c in DETECTORS]].median()\n", "print((catch * 100).round(1).to_string())\n", "print(\"\\nmedian days to notice (blank = never noticed):\")\n", "print(lags.reindex([\"blip\", \"trend\", \"break\"]).to_string())" ] }, { "cell_type": "markdown", "id": "8a657ff4", "metadata": {}, "source": [ "### The step search, run as the days arrive\n", "\n", "The grid runs the step search once, at the end, over the last 90 days, and dates each\n", "boundary it finds. That says where the search puts a change once the days after it are in\n", "hand, not when it would have told you. Run it each day on the days seen so far, and the delay\n", "is the first day on which a boundary lands at or after the event. The quiet arm is the\n", "control: there, every boundary is a false alarm." ] }, { "cell_type": "code", "execution_count": 11, "id": "471843d3", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:21:43.378665Z", "iopub.status.busy": "2026-10-25T13:21:43.377317Z", "iopub.status.idle": "2026-10-25T13:23:07.397726Z", "shell.execute_reply": "2026-10-25T13:23:07.396189Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "step search run each day, break : fired on 88.3%, median 6.0 days after the event\n", "step search run each day, trend : fired on 97.2%, median 15.0 days after the event\n", "step search run each day, quiet : fired on 14.4%, median 14.0 days after the event\n", "Read back at the end, the same search dates the step 0 days after it began; run as the days arrive, it needs 6 days of the new level to say so.\n" ] } ], "source": [ "def live_first_fire(v, d, lo=EVENT_DAY, hi=EVENT_DAY + FOLLOW):\n", " \"\"\"Days from lo until a search run on the days seen so far first places a boundary at or after lo.\"\"\"\n", " for t in range(lo, hi):\n", " if any(lo <= c <= t for c in break_days(v[:t + 1], d[:t + 1], penalty_sd=PENALTY)):\n", " return t - lo\n", " return None\n", "\n", "\n", "live_rows = []\n", "for shape in [\"quiet\", \"trend\", \"break\"]:\n", " for rep in range(N_REPS):\n", " g = np.random.default_rng(SHAPE_SEED[shape] + rep)\n", " df = make_series(shape, SHAPES[shape], generator=g)\n", " lag = live_first_fire(df[\"value\"].to_numpy(), df[\"dow\"].to_numpy())\n", " live_rows.append({\"shape\": shape, \"caught\": lag is not None, \"lag\": lag})\n", "live = pd.DataFrame(live_rows)\n", "live_step = {}\n", "for shape in [\"quiet\", \"trend\", \"break\"]:\n", " rows = live[live[\"shape\"] == shape]\n", " lag_vals = rows[\"lag\"].dropna()\n", " live_step[shape] = {\"caught\": round(float(rows[\"caught\"].mean()), 4),\n", " \"median_lag_days\": None if lag_vals.empty else float(lag_vals.median())}\n", "for shape in [\"break\", \"trend\", \"quiet\"]:\n", " print(f\"step search run each day, {shape:6s}: fired on {live_step[shape]['caught'] * 100:4.1f}%, \"\n", " f\"median {live_step[shape]['median_lag_days']} days after the event\")\n", "print(\"Read back at the end, the same search dates the step \"\n", " f\"{lags.loc['break', 'segmentation lag']:.0f} days after it began; run as the days arrive, it \"\n", " f\"needs {live_step['break']['median_lag_days']:.0f} days of the new level to say so.\")" ] }, { "cell_type": "code", "execution_count": 12, "id": "55e2e725", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:23:07.407567Z", "iopub.status.busy": "2026-10-25T13:23:07.406290Z", "iopub.status.idle": "2026-10-25T13:23:07.816969Z", "shell.execute_reply": "2026-10-25T13:23:07.814666Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "wrote fig_grid.png\n" ] } ], "source": [ "fig, ax = plt.subplots(figsize=(8.6, 4.4))\n", "mat = catch.to_numpy() * 100\n", "ax.imshow(mat, cmap=\"BuGn\", vmin=0, vmax=100, aspect=\"auto\")\n", "ax.set_xticks(range(len(DETECTORS)), DETECTORS)\n", "ax.set_yticks(range(4), [i if i != \"quiet\" else \"quiet (nothing happened)\"\n", " for i in catch.index])\n", "ax.set_xlabel(\"detector run, all at the same false-alarm cost\")\n", "ax.set_ylabel(\"what was actually planted\")\n", "for i in range(mat.shape[0]):\n", " for j in range(mat.shape[1]):\n", " ax.text(j, i, f\"{mat[i, j]:.0f}%\", ha=\"center\", va=\"center\",\n", " color=INK if mat[i, j] < 55 else PAPER, fontsize=11)\n", "ax.grid(False)\n", "ax.set_title(\"Each detector is close to blind to the shapes it was not built for\",\n", " loc=\"left\", fontsize=12, color=INK)\n", "fig.tight_layout()\n", "fig.savefig(FIGURES / \"fig_grid.png\", bbox_inches=\"tight\", dpi=150)\n", "plt.close(fig)\n", "print(\"wrote fig_grid.png\")" ] }, { "cell_type": "markdown", "id": "221bf60a", "metadata": {}, "source": [ "### The quiet tracked metric is the control, and a zero there still needs a margin\n", "\n", "The control shape is the one to read hardest. Nothing was planted in those tracked metrics, so every\n", "fire there is a false alarm - and it is the arm that proves the calibration did what it\n", "claimed rather than the arm that proves the detectors are good.\n", "\n", "When a detector fires on none of the quiet tracked metrics, the honest statement is not \"this never\n", "false-alarms\". It is \"I saw none in this many tracked metrics, which is consistent with a true rate\n", "anywhere up to the bound below\". With no events in $n$ trials, the upper end of the 95%\n", "interval is about $3/n$ - the rule of three." ] }, { "cell_type": "code", "execution_count": 13, "id": "ec8a9341", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:23:07.826978Z", "iopub.status.busy": "2026-10-25T13:23:07.825174Z", "iopub.status.idle": "2026-10-25T13:23:07.846204Z", "shell.execute_reply": "2026-10-25T13:23:07.843861Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "textbook one-day : 17 of 180 quiet tracked metrics fired over 28 days = 9.4%\n", "robust one-day : 17 of 180 quiet tracked metrics fired over 28 days = 9.4%\n", "trend : 6 of 180 quiet tracked metrics fired over 28 days = 3.3%\n", "segmentation : 10 of 180 quiet tracked metrics fired over 28 days = 5.6%\n" ] } ], "source": [ "def zero_margin(n):\n", " \"\"\"Upper 95% bound on a rate when zero events were observed in n trials.\"\"\"\n", " return 3.0 / n\n", "\n", "\n", "quiet = grid[grid[\"shape\"] == \"quiet\"]\n", "false_alarm = {}\n", "for col in DETECTORS:\n", " hits, n = int(quiet[col].sum()), len(quiet)\n", " false_alarm[col] = {\"fired_on\": hits, \"of_rows\": n, \"rate\": hits / n,\n", " \"upper95\": zero_margin(n) if hits == 0 else None}\n", " if hits == 0:\n", " print(f\"{col:18s}: 0 of {n} quiet tracked metrics - true rate could still be as high as \"\n", " f\"{zero_margin(n) * 100:.1f}%\")\n", " else:\n", " print(f\"{col:18s}: {hits} of {n} quiet tracked metrics fired over {FOLLOW} days \"\n", " f\"= {hits / n * 100:.1f}%\")" ] }, { "cell_type": "markdown", "id": "fc4a5ed4", "metadata": {}, "source": [ "## How big does it have to be before you can see it at all?\n", "\n", "The grid says which detector suits which shape. It does not say the thing a marketer\n", "actually needs, which is: **how large does a change have to be before my data can show\n", "it to me?** That number is not universal. It depends on how noisy the account is, how\n", "many metrics you track, and how long you are willing to wait - all of which are properties\n", "of your account, not of the method.\n", "\n", "So here it is as a curve rather than a claim. Hold the false-alarm budget fixed, vary\n", "the size of the planted event, and read off how often it gets caught. Where the curve\n", "crosses the halfway line is the honest answer to \"would I have noticed?\"\n", "\n", "One thing this makes plain. Below a certain size, a one-day excursion cannot be\n", "separated from the ordinary glitches every account produces - not because the detector\n", "is weak, but because a bad day and a bad day look the same. Telling those apart is a\n", "question about *causes*, and no amount of statistics on one series answers it. The\n", "level change is the one that stays detectable, and that is the useful thing to know." ] }, { "cell_type": "code", "execution_count": 14, "id": "07e6c4d4", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:23:07.851225Z", "iopub.status.busy": "2026-10-25T13:23:07.850821Z", "iopub.status.idle": "2026-10-25T13:23:33.459301Z", "shell.execute_reply": "2026-10-25T13:23:33.457334Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " kind size caught\n", " one bad day 40 11\n", " one bad day 80 38\n", " one bad day 120 70\n", " one bad day 180 100\n", " one bad day 260 100\n", "a step to a new level 6 14\n", "a step to a new level 10 33\n", "a step to a new level 16 68\n", "a step to a new level 24 82\n", "a step to a new level 34 92\n", "\n", "caught half the time at: one bad day +95%, a step +13%\n" ] } ], "source": [ "LIMIT_REPS = 100\n", "BLIP_SIZES = [0.4, 0.8, 1.2, 1.8, 2.6]\n", "BREAK_SIZES = [0.06, 0.10, 0.16, 0.24, 0.34]\n", "\n", "limit_rows = []\n", "for mag in BLIP_SIZES:\n", " caught = 0\n", " for rep in range(LIMIT_REPS):\n", " g = np.random.default_rng(83_000 + rep)\n", " df = make_series(\"blip\", mag, generator=g)\n", " v, d = df[\"value\"].to_numpy(), df[\"dow\"].to_numpy()\n", " rz = np.abs(np.nan_to_num(robust_scores(v, d), nan=0.0))\n", " caught += int((rz[EVENT_DAY:EVENT_DAY + FOLLOW] > CUT_ROBUST).any())\n", " limit_rows.append({\"kind\": \"one bad day\", \"size\": mag, \"caught\": caught / LIMIT_REPS})\n", "\n", "for mag in BREAK_SIZES:\n", " caught = 0\n", " for rep in range(LIMIT_REPS):\n", " g = np.random.default_rng(84_000 + rep)\n", " df = make_series(\"break\", mag, generator=g)\n", " v, d = df[\"value\"].to_numpy(), df[\"dow\"].to_numpy()\n", " caught += int(any(EVENT_DAY <= c < EVENT_DAY + FOLLOW\n", " for c in break_days(v, d, penalty_sd=PENALTY)))\n", " limit_rows.append({\"kind\": \"a step to a new level\", \"size\": mag,\n", " \"caught\": caught / LIMIT_REPS})\n", "\n", "limits = pd.DataFrame(limit_rows)\n", "print(limits.assign(size=lambda t: (t[\"size\"] * 100).round(0).astype(int),\n", " caught=lambda t: (t[\"caught\"] * 100).round(0).astype(int))\n", " .to_string(index=False))\n", "\n", "\n", "def half_catch_size(block):\n", " \"\"\"Interpolate the event size at which the detector catches half the time.\"\"\"\n", " b = block.sort_values(\"size\")\n", " xs, ys = b[\"size\"].to_numpy(), b[\"caught\"].to_numpy()\n", " if ys.max() < 0.5 or ys.min() > 0.5:\n", " return None\n", " return float(np.interp(0.5, ys, xs))\n", "\n", "\n", "half_blip = half_catch_size(limits[limits[\"kind\"] == \"one bad day\"])\n", "half_break = half_catch_size(limits[limits[\"kind\"] == \"a step to a new level\"])\n", "print(f\"\\ncaught half the time at: one bad day \"\n", " f\"{'n/a' if half_blip is None else f'+{half_blip * 100:.0f}%'}, \"\n", " f\"a step {'n/a' if half_break is None else f'+{half_break * 100:.0f}%'}\")" ] }, { "cell_type": "code", "execution_count": 15, "id": "44c4cddb", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:23:33.469183Z", "iopub.status.busy": "2026-10-25T13:23:33.468604Z", "iopub.status.idle": "2026-10-25T13:23:33.732953Z", "shell.execute_reply": "2026-10-25T13:23:33.732261Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "wrote fig_limits.png\n" ] } ], "source": [ "fig, ax = plt.subplots(figsize=(8.6, 4.4))\n", "for kind, colour in [(\"one bad day\", FLAG), (\"a step to a new level\", NAIVE)]:\n", " b = limits[limits[\"kind\"] == kind].sort_values(\"size\")\n", " ax.plot(b[\"size\"] * 100, b[\"caught\"] * 100, color=colour, lw=2.2, marker=\"o\",\n", " label=kind)\n", "ax.axhline(50, color=MUTED, lw=1.0, ls=\"--\")\n", "ax.text(ax.get_xlim()[1], 52, \"caught half the time\", color=MUTED, fontsize=9, ha=\"right\")\n", "ax.set_xlabel(\"size of the change, % of the normal level\")\n", "ax.set_ylabel(\"caught within four weeks, %\")\n", "ax.set_ylim(0, 100)\n", "ax.set_title(\"A step shows up at a fraction of the size a single bad day needs\",\n", " loc=\"left\", fontsize=12, color=INK)\n", "ax.legend(frameon=False, fontsize=9, loc=\"lower right\")\n", "fig.tight_layout()\n", "fig.savefig(FIGURES / \"fig_limits.png\", bbox_inches=\"tight\", dpi=150)\n", "plt.close(fig)\n", "print(\"wrote fig_limits.png\")" ] }, { "cell_type": "markdown", "id": "cdba2ebd", "metadata": {}, "source": [ "## What the alarms cost across a whole account\n", "\n", "Every rate above is per tracked metric. An account is not one tracked metric. An advertiser\n", "alarming on cost per acquisition, click-through rate, conversion rate and spend across the\n", "campaigns that matter is watching about twenty tracked metrics, every day.\n", "\n", "Take a per-metric-per-day false-alarm rate $p$. Across $R$ tracked metrics over $D$ days the number of\n", "false alarms to expect is\n", "\n", "$$ \\mathbb{E}[\\text{false alarms}] = p \\cdot R \\cdot D $$\n", "\n", "and the chance of getting through a day with none at all is $(1-p)^{R}$.\n", "\n", "This is the same arithmetic as scanning a breakdown table and picking out the biggest\n", "gap: doing it once is a fair question, doing it twenty times a day is a search. The\n", "consequence is the thing to take away from this post. **The alarm line is not a\n", "statistical constant you look up. It is a budget you choose.** Decide how many false\n", "alarms a week your team will really investigate before they start ignoring the alerts,\n", "then read the line off that." ] }, { "cell_type": "code", "execution_count": 16, "id": "1bfe45de", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:23:33.740539Z", "iopub.status.busy": "2026-10-25T13:23:33.740150Z", "iopub.status.idle": "2026-10-25T13:23:33.765131Z", "shell.execute_reply": "2026-10-25T13:23:33.761644Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " false alarms per tracked metric per year robust score line per week, account chance of a quiet day\n", " 52 2.35 19.9 0.0462\n", " 12 4.29 4.6 0.5124\n", " 4 7.86 1.5 0.8022\n", " 1 12.30 0.4 0.9466\n" ] } ], "source": [ "POOL = np.concatenate([np.abs(p[~np.isnan(p)]) for p in Q_ROBUST])\n", "# How many things an account actually alarms on. Not measured - stated, because it is a property\n", "# of the reader's setup rather than of this data. A handful of KPIs per platform (conversion rate,\n", "# cost per acquisition, cost per click, click-through rate) across the campaigns that matter. The\n", "# deck used to say two hundred, which is not an account anybody runs.\n", "ACCOUNT_METRICS = 20\n", "\n", "budget_rows = []\n", "for per_year in [52, 12, 4, 1]:\n", " p = per_year / 365.0\n", " budget_rows.append({\n", " \"false alarms per tracked metric per year\": per_year,\n", " \"robust score line\": round(float(np.quantile(POOL, 1 - p)), 2),\n", " \"per week, account\": round(p * ACCOUNT_METRICS * 7, 1),\n", " \"chance of a quiet day\": round(float((1 - p) ** ACCOUNT_METRICS), 4),\n", " })\n", "budget = pd.DataFrame(budget_rows)\n", "print(budget.to_string(index=False))" ] }, { "cell_type": "code", "execution_count": 17, "id": "ed5246ea", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:23:33.772009Z", "iopub.status.busy": "2026-10-25T13:23:33.771341Z", "iopub.status.idle": "2026-10-25T13:23:34.058234Z", "shell.execute_reply": "2026-10-25T13:23:34.056719Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "wrote fig_budget.png\n" ] } ], "source": [ "fig, ax = plt.subplots(figsize=(8.5, 4.4))\n", "ax.plot(budget[\"robust score line\"], budget[\"per week, account\"],\n", " color=FLAG, lw=2.2, marker=\"o\")\n", "ax.set_xlabel(\"where you put the alarm line (robust score)\")\n", "ax.set_ylabel(\"false alarms per week\")\n", "ax.set_title(f\"What a threshold costs an account of {ACCOUNT_METRICS} tracked metrics (synthetic)\",\n", " loc=\"left\", fontsize=12, color=INK)\n", "for _, r in budget.iterrows():\n", " ax.annotate(f\"{r['per week, account']:.1f}/wk\",\n", " (r[\"robust score line\"], r[\"per week, account\"]),\n", " textcoords=\"offset points\", xytext=(0, 9), ha=\"center\",\n", " fontsize=9, color=MUTED)\n", "fig.tight_layout()\n", "fig.savefig(FIGURES / \"fig_budget.png\", bbox_inches=\"tight\", dpi=150)\n", "plt.close(fig)\n", "print(\"wrote fig_budget.png\")" ] }, { "cell_type": "markdown", "id": "c282740a", "metadata": {}, "source": [ "## Catching something is not the same as describing it\n", "\n", "One number in the grid above deserves suspicion rather than applause: the segmentation\n", "detector notices essentially every planted trend. It was built to find steps, so why is\n", "it perfect on slides?\n", "\n", "Because it does the wrong thing very reliably. Given a smooth ramp, a method whose only\n", "vocabulary is \"the level changed on this date\" describes it as a step, usually one, in the\n", "middle of the ramp - a step that never happened. It fires, so it scores as a catch; what it\n", "hands the operator is a date on which nothing in particular occurred.\n", "\n", "This is the failure mode the algorithm's own literature names, and it is worth measuring\n", "rather than trusting: count how many boundaries each shape provokes. A real step should\n", "produce one. A slide should produce one only if the detector is being honest about not\n", "knowing." ] }, { "cell_type": "code", "execution_count": 18, "id": "6ba4f2bd", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:23:34.066443Z", "iopub.status.busy": "2026-10-25T13:23:34.065980Z", "iopub.status.idle": "2026-10-25T13:23:43.553754Z", "shell.execute_reply": "2026-10-25T13:23:43.552207Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "shape mean_boundaries share_2_or_more\n", "quiet 0.058 0.010\n", "break 0.820 0.048\n", "trend 1.055 0.070\n", "\n", "a real step produces 0.82 boundaries on average; a smooth slide produces 1.05; two or more on 7% of slides and 5% of steps\n" ] } ], "source": [ "CUTCOUNT_REPS = 400\n", "cutcount_rows = []\n", "for shape, mag in [(\"quiet\", 0.0), (\"break\", 0.22), (\"trend\", 0.30)]:\n", " counts = []\n", " for rep in range(CUTCOUNT_REPS):\n", " g = np.random.default_rng(77_000 + rep)\n", " df = make_series(shape, mag, generator=g)\n", " v, d = df[\"value\"].to_numpy(), df[\"dow\"].to_numpy()\n", " cuts = break_days(v, d, penalty_sd=PENALTY)\n", " counts.append(sum(1 for c in cuts if EVENT_DAY <= c < EVENT_DAY + FOLLOW))\n", " cutcount_rows.append({\n", " \"shape\": shape,\n", " \"mean_boundaries\": float(np.mean(counts)),\n", " \"share_2_or_more\": float(np.mean(np.array(counts) >= 2)),\n", " })\n", "cutcounts = pd.DataFrame(cutcount_rows)\n", "print(cutcounts.round(3).to_string(index=False))\n", "\n", "trend_boundaries = float(cutcounts.loc[cutcounts[\"shape\"] == \"trend\",\n", " \"mean_boundaries\"].iloc[0])\n", "break_boundaries = float(cutcounts.loc[cutcounts[\"shape\"] == \"break\",\n", " \"mean_boundaries\"].iloc[0])\n", "print(f\"\\na real step produces {break_boundaries:.2f} boundaries on average; \"\n", " f\"a smooth slide produces {trend_boundaries:.2f}; two or more on \"\n", " f\"{cutcounts.loc[cutcounts['shape'] == 'trend', 'share_2_or_more'].iloc[0] * 100:.0f}% of slides \"\n", " f\"and {cutcounts.loc[cutcounts['shape'] == 'break', 'share_2_or_more'].iloc[0] * 100:.0f}% of steps\")" ] }, { "cell_type": "markdown", "id": "a9f2eb0b", "metadata": {}, "source": [ "## The newest days are the emptiest, and that breaks the day test\n", "\n", "Everything above scores today against the last four weeks. On a conversion metric that\n", "is unsound, and it is worth being blunt about why.\n", "\n", "Conversions arrive late. Somebody clicks today and converts on Thursday, so the count\n", "attached to today keeps rising for days after today is over. Cost, meanwhile, is\n", "complete immediately. A cost-per-acquisition computed on the run date is therefore\n", "**cost divided by a conversion count that has not finished arriving** - it reads high,\n", "and it reads high every single day.\n", "\n", "A detector that compares that against four weeks of finished days does not find a\n", "problem. It finds the calendar. And because the distortion is in the adverse direction\n", "every time, it fires adversely every time, which is the worst possible failure for an\n", "alarm: constant, one-directional, and completely fake.\n", "\n", "Below is the size of it. The same quiet tracked metrics as before - nothing planted, nothing wrong\n", "- with an ordinary reporting curve applied to the last fortnight: about a third of a\n", "day's conversions in on the day itself, most of them by the end of the week, effectively\n", "all of them after two weeks." ] }, { "cell_type": "code", "execution_count": 19, "id": "c6d257b1", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:23:43.559268Z", "iopub.status.busy": "2026-10-25T13:23:43.558850Z", "iopub.status.idle": "2026-10-25T13:24:25.420752Z", "shell.execute_reply": "2026-10-25T13:24:25.419279Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "quiet tracked metrics - nothing planted, nothing wrong. Share of tracked metrics that fire:\n", "\n", " day slope level\n", " no reporting lag, scored today 0.7% 0.0% 0.7%\n", " flat-curve control, scored today 0.7% 0.0% 0.7%\n", " WITH the lag, scored today 62.7% 30.0% 99.3%\n", " with the lag, scored on a matured window 0.7% 0.0% 3.3%\n", "\n", " control arm reproduces the no-lag arm exactly: True\n", " median day score on the run date under lag: 13.6 against an alarm line of 12.3\n" ] } ], "source": [ "# The converter-conditional reported fraction: the share of a day's eventual conversions\n", "# that have arrived by the time you look, indexed by how many days ago that day was. A\n", "# realistic shape for a considered purchase; a shorter cycle fills in faster and a longer\n", "# one slower, which is the point - it is a property of the account, not a constant.\n", "REPORTED_BY_DAY = np.array([0.35, 0.55, 0.70, 0.80, 0.87, 0.92, 0.95, 0.97,\n", " 0.98, 0.99, 0.995, 1.0, 1.0, 1.0])\n", "\n", "\n", "def apply_reporting_lag(values, curve=REPORTED_BY_DAY):\n", " \"\"\"Inflate the most recent days the way an unfinished conversion count does.\n", "\n", " Cost is complete on the day; conversions are not. So a cost-per-acquisition on the\n", " run date is cost over a partial count, and it reads HIGH - the adverse direction for\n", " that metric, every day, which is what makes the resulting alarm so convincing.\n", " \"\"\"\n", " out = np.asarray(values, dtype=float).copy()\n", " n = len(out)\n", " for ago, frac in enumerate(curve):\n", " idx = n - 1 - ago\n", " if idx >= 0 and frac > 0:\n", " out[idx] = out[idx] / frac\n", " return out\n", "\n", "\n", "LAG_REPS = 150\n", "SETTLED_BACK = len(REPORTED_BY_DAY) - 1 # the freshest day that has finished arriving\n", "\n", "\n", "def flat_curve_control(values):\n", " \"\"\"The control arm: the same machinery, a curve that distorts nothing.\n", "\n", " A no-lag arm on its own leaves a hole - it varies the code path as well as the\n", " curve. Running a curve of ones through the SAME function holds everything fixed\n", " except the shape being tested, so a difference can only be the shape. It must\n", " reproduce the no-lag arm exactly; where it does not, the finding is about the\n", " harness rather than about reporting lag.\n", " \"\"\"\n", " return apply_reporting_lag(values, curve=np.ones(len(REPORTED_BY_DAY)))\n", "\n", "\n", "lag_rows = []\n", "for rep in range(LAG_REPS):\n", " g = np.random.default_rng(91_000 + rep)\n", " df = make_series(\"quiet\", 0.0, generator=g)\n", " v_clean, d = df[\"value\"].to_numpy(), df[\"dow\"].to_numpy()\n", " v_flat = flat_curve_control(v_clean)\n", " v_lagged = apply_reporting_lag(v_clean)\n", " last = len(v_clean) - 1\n", " settled = last - SETTLED_BACK\n", "\n", " row = {}\n", " for tag, v in ((\"clean\", v_clean), (\"flat\", v_flat), (\"lagged\", v_lagged)):\n", " rz, tz = robust_scores(v, d), trend_scores(v, d)\n", " bd = break_days(v, d, penalty_sd=PENALTY)\n", " row[tag + \"_day\"] = bool(abs(rz[last]) > CUT_ROBUST)\n", " row[tag + \"_slope\"] = bool(not np.isnan(tz[last]) and abs(tz[last]) > CUT_TREND)\n", " row[tag + \"_level\"] = bool(any(c > last - len(REPORTED_BY_DAY) for c in bd))\n", " if tag == \"lagged\":\n", " row[\"lagged_day_score\"] = float(abs(rz[last]))\n", " # the same three tests, scored on a window that has finished arriving\n", " rzs, tzs = rz[settled], tz[settled]\n", " row[\"settled_day\"] = bool(abs(rzs) > CUT_ROBUST)\n", " row[\"settled_slope\"] = bool(not np.isnan(tzs) and abs(tzs) > CUT_TREND)\n", " # the day and slope scores only look back, but the step search sees the whole series,\n", " # so the settled read runs it on the days that have finished arriving and no others\n", " bds = break_days(v[:settled + 1], d[:settled + 1], penalty_sd=PENALTY)\n", " row[\"settled_level\"] = bool(any(c > settled - len(REPORTED_BY_DAY) for c in bds))\n", " lag_rows.append(row)\n", "lag = pd.DataFrame(lag_rows)\n", "\n", "LAG_ARMS = [\"clean\", \"flat\", \"lagged\", \"settled\"]\n", "LAG_TESTS = [\"day\", \"slope\", \"level\"]\n", "lag_table = {arm: {t: float(lag[arm + \"_\" + t].mean()) for t in LAG_TESTS}\n", " for arm in LAG_ARMS}\n", "lag_median_score = float(lag[\"lagged_day_score\"].median())\n", "\n", "lag_clean_rate = lag_table[\"clean\"][\"day\"]\n", "lag_naive_rate = lag_table[\"lagged\"][\"day\"]\n", "lag_settled_rate = lag_table[\"settled\"][\"day\"]\n", "\n", "control_holds = all(\n", " abs(lag_table[\"clean\"][t] - lag_table[\"flat\"][t]) < 1e-9 for t in LAG_TESTS)\n", "\n", "LAG_LABELS = {\n", " \"clean\": \"no reporting lag, scored today\",\n", " \"flat\": \"flat-curve control, scored today\",\n", " \"lagged\": \"WITH the lag, scored today\",\n", " \"settled\": \"with the lag, scored on a matured window\",\n", "}\n", "print(\"quiet tracked metrics - nothing planted, nothing wrong. Share of tracked metrics that fire:\")\n", "print()\n", "print(\" {:41s} {:>7s} {:>7s} {:>7s}\".format(\"\", \"day\", \"slope\", \"level\"))\n", "for arm in LAG_ARMS:\n", " r = lag_table[arm]\n", " print(\" {:41s} {:6.1f}% {:6.1f}% {:6.1f}%\".format(\n", " LAG_LABELS[arm], r[\"day\"] * 100, r[\"slope\"] * 100, r[\"level\"] * 100))\n", "print()\n", "print(\" control arm reproduces the no-lag arm exactly: \" + str(control_holds))\n", "print(\" median day score on the run date under lag: \"\n", " \"{:.1f} against an alarm line of {:.1f}\".format(lag_median_score, CUT_ROBUST))\n", "if not control_holds:\n", " print(\" WARNING: it does not, so nothing below is attributable to the curve alone.\")" ] }, { "cell_type": "markdown", "id": "0521b88a", "metadata": {}, "source": [ "Read the middle line again. Nothing happened in any of those tracked metrics. The detector is not\n", "wrong about the arithmetic - the number really is high - it is wrong about what the\n", "number means, and no amount of calibration fixes it, because the calibration was done\n", "on tracked metrics that had the same distortion or on tracked metrics that had none.\n", "\n", "**The fix is not statistical, it is a rule about which day you score.** Stop scoring days\n", "whose counts have not finished arriving. The bottom line above is the same metrics, the\n", "same lag, scored on the most recent day that has settled instead of on today - and it\n", "lands back where the calibration promised.\n", "\n", "What it costs you is the wait. On this reporting curve you are looking at a day about\n", "two weeks old, which is a fortnight of blindness on the day test. That is the honest\n", "trade, and it is why the level tests matter more than the day test on any metric that\n", "reports late: a step change is still there in a fortnight, and a blip is not.\n", "\n", "How long your own account takes to settle is a question with its own method and its own\n", "post in this series - the one about why today's numbers always look bad. It fits a\n", "survival curve to your click-to-conversion times and reports the share reported by day\n", "N, which is exactly the curve assumed above. Do not guess it; measure it." ] }, { "cell_type": "code", "execution_count": 20, "id": "fd6968a7", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:24:25.430619Z", "iopub.status.busy": "2026-10-25T13:24:25.424743Z", "iopub.status.idle": "2026-10-25T13:24:25.799751Z", "shell.execute_reply": "2026-10-25T13:24:25.798620Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "wrote fig_lag.png\n" ] } ], "source": [ "fig, ax = plt.subplots(figsize=(8.6, 4.4))\n", "xs = np.arange(len(REPORTED_BY_DAY))\n", "ax.plot(xs, REPORTED_BY_DAY * 100, color=NAIVE, lw=2.4, marker=\"o\", ms=6)\n", "ax.axhline(100, color=LINE, lw=1.2)\n", "ax.set_xlabel(\"days since the day in question\")\n", "ax.set_ylabel(\"share of its conversions in, %\")\n", "ax.set_ylim(0, 108)\n", "ax.set_title(\"Why the run date cannot be scored: its count has not finished arriving\",\n", " loc=\"left\", fontsize=12, color=INK)\n", "ax.annotate(f\"scoring here fires on {lag_naive_rate * 100:.0f}% of QUIET metrics\",\n", " xy=(0, REPORTED_BY_DAY[0] * 100), xytext=(1.6, 46),\n", " arrowprops=dict(arrowstyle=\"-|>\", color=FLAG, lw=1.6),\n", " color=FLAG, fontsize=11.5)\n", "ax.annotate(f\"scoring here fires on {lag_settled_rate * 100:.0f}%\",\n", " xy=(len(REPORTED_BY_DAY) - 1, 100), xytext=(7.4, 66),\n", " arrowprops=dict(arrowstyle=\"-|>\", color=SIGNAL, lw=1.6),\n", " color=SIGNAL, fontsize=11.5)\n", "fig.tight_layout()\n", "fig.savefig(FIGURES / \"fig_lag.png\", bbox_inches=\"tight\", dpi=150)\n", "plt.close(fig)\n", "print(\"wrote fig_lag.png\")" ] }, { "cell_type": "markdown", "id": "b5b9b229", "metadata": {}, "source": [ "## Three things the step search gets wrong\n", "\n", "The step search assumes a series sits on one level, then jumps to another and sits there.\n", "Three kinds of series break that picture without any data being wrong, and each is measured\n", "here on quiet synthetic series with the shape planted on top, the step search run as it is\n", "everywhere above (its last 90 days, the calibrated penalty).\n", "\n", "**Two steps close together.** A rise of 15 per cent, and five days later another of 15 per cent.\n", "**A turn with no jump.** The level climbs for 40 days and then falls back, with no step anywhere.\n", "**Days that echo each other more.** Nothing planted, but today leans on yesterday more strongly\n", "than the 0.45 the rest of this notebook uses." ] }, { "cell_type": "code", "execution_count": 21, "id": "e0a454ae", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:24:25.807157Z", "iopub.status.busy": "2026-10-25T13:24:25.806435Z", "iopub.status.idle": "2026-10-25T13:24:38.414693Z", "shell.execute_reply": "2026-10-25T13:24:38.413020Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "two 15% steps 5 days apart, read as one step: 85.8%\n", "a turn with no jump, a boundary within 5 days of the turn: 8.0%; two or more boundaries on the ramps: 57.0%\n", "quiet series, a false step at echo 0.45: 10.5%; at echo 0.7: 68.5%\n" ] } ], "source": [ "SEG_REPS = 400\n", "SEG_STEP = 0.15\n", "SEG_GAP = 5\n", "SEG_TURN_DAY = 160\n", "SEG_ECHO = 0.7\n", "\n", "\n", "def seg_boundaries(kind, rho=RHO, reps=SEG_REPS):\n", " \"\"\"Boundaries the step search returns on each of `reps` quiet series with `kind` planted.\"\"\"\n", " out = []\n", " for r in range(reps):\n", " df = make_series(\"quiet\", 0.0, rho=rho, generator=np.random.default_rng(SEED + 5000 + r))\n", " v, day = df[\"value\"].to_numpy().copy(), df[\"day\"].to_numpy()\n", " if kind == \"two_close\":\n", " v *= np.where(day >= EVENT_DAY, 1 + SEG_STEP, 1.0)\n", " v *= np.where(day >= EVENT_DAY + SEG_GAP, 1 + SEG_STEP, 1.0)\n", " elif kind == \"turn\":\n", " up = np.clip(day - (SEG_TURN_DAY - 40), 0, 40)\n", " down = np.clip(day - SEG_TURN_DAY, 0, None)\n", " v *= 1 + 0.006 * (up - down)\n", " out.append(break_days(v, df[\"dow\"].to_numpy(), penalty_sd=PENALTY))\n", " return out\n", "\n", "\n", "_two = seg_boundaries(\"two_close\")\n", "_turn = seg_boundaries(\"turn\")\n", "seg_limits = {\n", " \"step\": SEG_STEP, \"gap_days\": SEG_GAP, \"echo\": SEG_ECHO, \"reps\": SEG_REPS,\n", " \"two_close_read_as_one\": float(np.mean([len(b) == 1 for b in _two])),\n", " \"turn_marked_near_turn\": float(np.mean([any(abs(c - SEG_TURN_DAY) <= 5 for c in b) for b in _turn])),\n", " \"turn_two_or_more\": float(np.mean([len(b) >= 2 for b in _turn])),\n", " \"quiet_false_step_usual_echo\": float(np.mean([len(b) > 0 for b in seg_boundaries(\"quiet\")])),\n", " \"quiet_false_step_strong_echo\": float(np.mean([len(b) > 0 for b in seg_boundaries(\"quiet\", rho=SEG_ECHO)])),\n", "}\n", "print(f\"two {SEG_STEP:.0%} steps {SEG_GAP} days apart, read as one step: \"\n", " f\"{seg_limits['two_close_read_as_one']:.1%}\")\n", "print(f\"a turn with no jump, a boundary within 5 days of the turn: {seg_limits['turn_marked_near_turn']:.1%}\"\n", " f\"; two or more boundaries on the ramps: {seg_limits['turn_two_or_more']:.1%}\")\n", "print(f\"quiet series, a false step at echo {RHO}: {seg_limits['quiet_false_step_usual_echo']:.1%}\"\n", " f\"; at echo {SEG_ECHO}: {seg_limits['quiet_false_step_strong_echo']:.1%}\")" ] }, { "cell_type": "markdown", "id": "a7b21520", "metadata": {}, "source": [ "Two steps close together come back as one bigger step on a date in between, so the date you\n", "check against your change log can be neither change. A turn is mostly not marked where it\n", "happened: the search cuts the ramps into pieces instead, which is the slide problem from\n", "earlier seen from the other side. And the penalty was calibrated on days that echo at 0.45;\n", "on a metric whose days echo more strongly, the same penalty finds steps in a quiet series\n", "most of the time. Recalibrate the penalty on your own quiet history, which measures your echo\n", "along with everything else." ] }, { "cell_type": "markdown", "id": "01425eae", "metadata": {}, "source": [ "## Everything the article quotes, written out\n", "\n", "Every number in the article, the widget and the video is read from the two files below.\n", "Nothing is typed by hand anywhere downstream." ] }, { "cell_type": "code", "execution_count": 22, "id": "e5b523f1", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:24:38.421048Z", "iopub.status.busy": "2026-10-25T13:24:38.419457Z", "iopub.status.idle": "2026-10-25T13:24:38.830458Z", "shell.execute_reply": "2026-10-25T13:24:38.829553Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "workbook case: day 150, sheet reading 15.16 against the notebook's 15.72\n", "wrote data/checks.json (4 pinned cases) and data/worked_example.json\n" ] } ], "source": [ "def series_payload(shape, mag, seed):\n", " df = make_series(shape, mag, generator=np.random.default_rng(seed))\n", " v_full, d = df[\"value\"].to_numpy(), df[\"dow\"].to_numpy()\n", "\n", " # Round FIRST, then score the rounded series. The widget only ever sees the stored\n", " # precision, so an expectation computed at full precision leaves the cross-check\n", " # passing on a rounding difference rather than on agreement - it measured a third\n", " # of the tolerance here before this line existed, which is a check one small edit\n", " # away from failing for a reason that is not a defect.\n", " v = np.round(v_full, 4)\n", " rz, tz = robust_scores(v, d), trend_scores(v, d)\n", "\n", " # The same series with the planted event removed. The shape branch consumes no\n", " # randomness, so re-drawing with the same seed reproduces this series' noise and\n", " # its glitches exactly - only the planted level differs. Storing it lets the\n", " # widget re-plant at any size the reader dials, from the notebook's own series\n", " # rather than from a regenerated lookalike.\n", " base = make_series(\"quiet\", 0.0, generator=np.random.default_rng(seed))[\"value\"].to_numpy()\n", " day = np.arange(len(v_full))\n", " if shape == \"blip\":\n", " level = np.where(day == EVENT_DAY, 1.0 + mag, 1.0)\n", " elif shape == \"trend\":\n", " level = 1.0 + mag * np.clip((day - EVENT_DAY) / 21.0, 0.0, 1.0)\n", " elif shape == \"break\":\n", " level = np.where(day >= EVENT_DAY, 1.0 + mag, 1.0)\n", " else:\n", " level = np.ones(len(v_full))\n", " assert np.allclose(base * level, v_full, rtol=1e-12), f\"base does not reconstruct {shape}\"\n", "\n", " return {\n", " \"shape\": shape,\n", " \"magnitude\": mag,\n", " \"seed\": seed,\n", " \"values\": [round(float(x), 4) for x in v],\n", " \"base_values\": [round(float(x), 6) for x in base],\n", " \"expected\": {\n", " \"max_robust_score\": round(\n", " float(np.nanmax(np.abs(rz[EVENT_DAY:EVENT_DAY + FOLLOW]))), 6),\n", " \"max_trend_score\": round(\n", " float(np.nanmax(np.abs(tz[EVENT_DAY:EVENT_DAY + FOLLOW]))), 6),\n", " \"break_days\": [int(x) for x in break_days(v, d, penalty_sd=PENALTY)],\n", " },\n", " }\n", "\n", "\n", "checks = {\n", " \"generated_by\": \"notebook.ipynb\",\n", " \"seed\": SEED,\n", " \"tolerance\": 1e-4,\n", " \"window\": WINDOW,\n", " \"cut_robust\": round(CUT_ROBUST, 6),\n", " \"cut_naive\": round(CUT_NAIVE, 6),\n", " \"cut_trend\": round(CUT_TREND, 6),\n", " \"penalty_sd\": PENALTY,\n", " \"follow_days\": FOLLOW,\n", " \"event_day\": EVENT_DAY,\n", " \"cases\": [\n", " series_payload(\"quiet\", 0.0, 1),\n", " series_payload(\"blip\", 1.20, 2),\n", " series_payload(\"trend\", 0.30, 3),\n", " series_payload(\"break\", 0.22, 4),\n", " ],\n", "}\n", "(DATA / \"checks.json\").write_text(json.dumps(checks, indent=2), encoding=\"utf-8\")\n", "\n", "# The workbook's target. A spreadsheet can carry this method's one-day score in full -\n", "# AVERAGEIF for the weekday means, MEDIAN and ABS for the spread - with exactly one\n", "# substitution: it fits the local trend with SLOPE(), the least-squares line, because\n", "# the median-of-pairwise-slopes fit needs a 378-term array formula that will not survive\n", "# a round trip through older spreadsheet software. That substitution is the sheet's one\n", "# simplification and it is the very thing the article warns about, so BOTH readings are\n", "# emitted here: the sheet has a true target to be checked against, and it can show the\n", "# reader what the robust fit would have said on the same 28 days.\n", "# The blip case on its own event day. A worked example has to SHOW the pipeline\n", "# producing an alarm, and the one-day score is the detector built for exactly this\n", "# shape - pointing the sheet at a step change instead gives a reader a silent verdict\n", "# and no idea whether the arithmetic worked.\n", "SHEET_SHAPE, SHEET_SEED, SHEET_MAG = \"blip\", 2, 1.20\n", "SHEET_DAY = EVENT_DAY\n", "\n", "_sv = np.round(make_series(SHEET_SHAPE, SHEET_MAG,\n", " generator=np.random.default_rng(SHEET_SEED))[\"value\"].to_numpy(), 4)\n", "_sd = np.arange(len(_sv)) % 7\n", "_base_v = _sv[SHEET_DAY - WINDOW:SHEET_DAY]\n", "_base_d = _sd[SHEET_DAY - WINDOW:SHEET_DAY]\n", "_means = dow_means(_base_v, _base_d)\n", "_adj = _base_v - _means[_base_d]\n", "_x = np.arange(WINDOW, dtype=float)\n", "\n", "\n", "def _sheet_reading(slope):\n", " \"\"\"One day's score, given whichever local-trend fit produced `slope`.\"\"\"\n", " resid = _adj - slope * _x\n", " med = float(np.median(resid))\n", " spread = float(np.median(np.abs(resid - med)))\n", " today = float(_sv[SHEET_DAY] - _means[_sd[SHEET_DAY]] - slope * WINDOW)\n", " return {\n", " \"slope\": float(slope),\n", " \"residuals\": [round(float(r), 6) for r in resid],\n", " \"median\": round(med, 6),\n", " \"mad\": round(spread, 6),\n", " \"today_residual\": round(today, 6),\n", " \"score\": round(0.6745 * (today - med) / spread, 6),\n", " }\n", "\n", "\n", "_ols = float(np.polyfit(_x, _adj, 1)[0])\n", "sheet_case = {\n", " \"shape\": SHEET_SHAPE, \"seed\": SHEET_SEED, \"magnitude\": SHEET_MAG,\n", " \"scored_day\": SHEET_DAY, \"window\": WINDOW, \"event_day\": EVENT_DAY,\n", " \"values\": [round(float(v), 4) for v in _sv[SHEET_DAY - WINDOW:SHEET_DAY + 1]],\n", " \"dow\": [int(d) for d in _sd[SHEET_DAY - WINDOW:SHEET_DAY + 1]],\n", " \"dow_means\": [round(float(m), 6) for m in _means],\n", " \"cut\": round(CUT_ROBUST, 4),\n", " \"sheet_fit_ols\": _sheet_reading(_ols),\n", " \"notebook_fit_robust\": _sheet_reading(theil_sen_slope(_adj)),\n", "}\n", "print(f\"workbook case: day {SHEET_DAY}, sheet reading \"\n", " f\"{sheet_case['sheet_fit_ols']['score']:.2f} against the notebook's \"\n", " f\"{sheet_case['notebook_fit_robust']['score']:.2f}\")\n", "\n", "worked = {\n", " \"generated_by\": \"notebook.ipynb\",\n", " \"spreadsheet_case\": sheet_case,\n", " \"data_label\": \"synthetic\",\n", " \"seed\": SEED,\n", " \"window_days\": WINDOW,\n", " \"follow_days\": FOLLOW,\n", " \"reps_per_shape\": N_REPS,\n", " \"autocorrelation\": RHO,\n", " \"planted\": dict(SHAPES),\n", " \"calibration\": {\n", " \"budget_per_row_per_year\": round(FALSE_ALARM_BUDGET * 365, 4),\n", " \"quiet_row_days\": int(n_robust),\n", " \"cut_robust\": round(CUT_ROBUST, 4),\n", " \"cut_naive\": round(CUT_NAIVE, 4),\n", " \"cut_trend\": round(CUT_TREND, 4),\n", " \"penalty_sd\": PENALTY,\n", " \"quiet_cuts_per_run\": round(quiet_cut_rate, 4),\n", " \"cut_allowance_per_run\": round(cut_allowance, 4),\n", " },\n", " \"catch_rate\": {\n", " shape: {col: round(float(catch.loc[shape, col]), 4) for col in catch.columns}\n", " for shape in catch.index\n", " },\n", " \"median_lag_days\": {\n", " shape: {col.replace(\" lag\", \"\"):\n", " (None if pd.isna(lags.loc[shape, col]) else float(lags.loc[shape, col]))\n", " for col in lags.columns}\n", " for shape in [\"blip\", \"trend\", \"break\"]\n", " },\n", " \"false_alarm\": false_alarm,\n", " \"step_search_live\": live_step,\n", " \"robustness\": {\n", " \"clean_gap_points\": round(clean_gap * 100, 2),\n", " \"dirty_gap_points\": round(dirty_gap * 100, 2),\n", " \"clean_gap_se_points\": round(clean_se * 100, 2),\n", " \"dirty_gap_se_points\": round(dirty_se * 100, 2),\n", " \"blip_size\": WORLD_BLIP,\n", " \"reps_per_world\": WORLD_REPS,\n", " \"spike_rate\": SPIKE_RATE,\n", " \"rows\": [\n", " {\"world\": r[\"world\"], \"shape\": r[\"shape\"],\n", " \"cut_robust\": float(r[\"cut_robust\"]), \"cut_textbook\": float(r[\"cut_textbook\"]),\n", " \"robust\": round(float(r[\"robust\"]), 4),\n", " \"textbook\": round(float(r[\"textbook\"]), 4)}\n", " for _, r in worlds.iterrows()\n", " ],\n", " },\n", " \"alarm_budget\": [\n", " {\"per_year\": int(r[\"false alarms per tracked metric per year\"]),\n", " \"robust_score_line\": float(r[\"robust score line\"]),\n", " \"per_week_account\": float(r[\"per week, account\"]),\n", " \"quiet_day_chance\": float(r[\"chance of a quiet day\"])}\n", " for _, r in budget.iterrows()\n", " ],\n", " \"money_example\": {\n", " # The two limits are shares of a normal level, and a share of an unstated level is not a\n", " # picture. So state the level: the median of the worked example's own baseline window,\n", " # rounded to whole currency because that is how the number gets said out loud.\n", " \"baseline\": round(float(np.median(_base_v))),\n", " \"step_level\": round(float(np.median(_base_v)) * (1 + half_break)),\n", " \"blip_level\": round(float(np.median(_base_v)) * (1 + half_blip)),\n", " },\n", " \"account_metrics\": ACCOUNT_METRICS,\n", " \"detection_limits\": {\n", " \"half_catch_blip\": None if half_blip is None else round(half_blip, 4),\n", " \"half_catch_break\": None if half_break is None else round(half_break, 4),\n", " \"rows\": [\n", " {\"kind\": r[\"kind\"], \"size\": float(r[\"size\"]),\n", " \"caught\": round(float(r[\"caught\"]), 4)}\n", " for _, r in limits.iterrows()\n", " ],\n", " },\n", " \"reporting_lag\": {\n", " \"curve\": [round(float(x), 4) for x in REPORTED_BY_DAY],\n", " \"settle_days\": len(REPORTED_BY_DAY) - 1,\n", " \"quiet_run_date_no_lag\": round(lag_clean_rate, 4),\n", " \"quiet_run_date_with_lag\": round(lag_naive_rate, 4),\n", " \"quiet_settled_day_with_lag\": round(lag_settled_rate, 4),\n", " \"by_test\": {arm: {t: round(v, 4) for t, v in row.items()}\n", " for arm, row in lag_table.items()},\n", " \"control_reproduces_no_lag\": bool(control_holds),\n", " \"median_run_date_score\": round(lag_median_score, 4),\n", " \"reps\": LAG_REPS,\n", " },\n", " \"segmentation_limits\": {k: (round(v, 4) if isinstance(v, float) else v)\n", " for k, v in seg_limits.items()},\n", " \"boundary_counts\": {\n", " \"trend_boundaries\": round(trend_boundaries, 4),\n", " \"break_boundaries\": round(break_boundaries, 4),\n", " \"rows\": [\n", " {\"shape\": r[\"shape\"],\n", " \"mean_boundaries\": round(float(r[\"mean_boundaries\"]), 4),\n", " \"share_2_or_more\": round(float(r[\"share_2_or_more\"]), 4)}\n", " for _, r in cutcounts.iterrows()\n", " ],\n", " },\n", "}\n", "(DATA / \"worked_example.json\").write_text(json.dumps(worked, indent=2), encoding=\"utf-8\")\n", "print(f\"wrote data/checks.json ({len(checks['cases'])} pinned cases) \"\n", " f\"and data/worked_example.json\")" ] }, { "cell_type": "markdown", "id": "61f37ca0", "metadata": {}, "source": [ "---\n", "\n", "# Run it on your own numbers\n", "\n", "Everything from here runs locally. Your file is read from your own disk, and nothing in\n", "this notebook sends it anywhere.\n", "\n", "## What the file needs to look like\n", "\n", "One entry per day for each metric you track - a tracked metric being one number you follow day by day,\n", "one campaign or one channel. Three columns:\n", "\n", "| column | what it holds | example |\n", "|---|---|---|\n", "| `date` | the day, any common format | `2026-04-17` |\n", "| `series` | the campaign or channel name - whatever you are tracking | `Brand - Search` |\n", "| `value` | the number for that day | `23.80` |\n", "\n", "One example entry: `2026-04-17,Brand - Search,23.80`\n", "\n", "**Where to get it.** In Google Ads: the Campaigns table, Segment, Time, Day, then\n", "Download as CSV (or .xlsx) - the title lines and Total rows are set aside for you. In GA4 the same shape comes out of\n", "the BigQuery export - one entry per day per campaign with your metric. If your export has\n", "cost and conversions in separate columns rather than a ready-made rate, the consolidator\n", "in the next cell builds the daily series for you.\n", "\n", "Set `MY_FILE` below to your file's path and run the rest." ] }, { "cell_type": "code", "execution_count": 23, "id": "e129e77f", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:24:38.836242Z", "iopub.status.busy": "2026-10-25T13:24:38.835803Z", "iopub.status.idle": "2026-10-25T13:24:39.107994Z", "shell.execute_reply": "2026-10-25T13:24:39.106797Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "loader defined - set MY_FILE above to use it\n" ] } ], "source": [ "MY_FILE = None # <-- put your CSV path here, e.g. \"my_campaigns.csv\"\n", "\n", "# HOW MANY TRAILING DAYS ARE STILL FILLING IN. On a conversion metric this is the single\n", "# most important setting on this page, and the default is deliberately not zero. Cost is\n", "# complete on the day; conversions are not, so a cost-per-acquisition on the run date is\n", "# cost over a partial count and reads adverse every single day. Scoring it finds the\n", "# calendar rather than a problem.\n", "#\n", "# Do not guess this. The sibling post on why today's numbers look bad fits it from your\n", "# own click-to-conversion times. Fourteen is a placeholder for a considered purchase; a\n", "# same-session ecommerce account settles in two or three, a long B2B cycle in far more.\n", "# Set it to 0 only for a metric that is complete on the day - spend, impressions, clicks.\n", "SETTLE_DAYS = 14\n", "MY_METRIC = \"value\" # or set to None and give COST_COL / CONV_COL below\n", "COST_COL = None # e.g. \"Cost\"\n", "CONV_COL = None # e.g. \"Conversions\"\n", "\n", "# ---- export reader (shared across every Stochastic Strata notebook; do not edit here) ----------\n", "# One reader for the file you bring. Ad platforms and analytics tools do not hand over clean tables:\n", "# a Google Ads download opens with a title line and a date-range line and closes with \"Total:\" rows,\n", "# a GA4 download opens with \"#\" lines and writes dates as 20261019, a European download separates\n", "# with semicolons and writes 1.234,56, and Excel turns numbers into \"1,234.56\" or \"€12.30\". This\n", "# reader finds the real header row, sets the title, comment and total rows aside, turns formatted\n", "# numbers back into numbers, and reads dates in the platforms' own forms. Where a file is genuinely\n", "# ambiguous - 06/07/2026 could be the 6th of July or the 7th of June - it stops and asks rather than\n", "# guessing, because a guessed date silently moves a day to the wrong weekday.\n", "#\n", "# table, notes = read_export(\"my_export.csv\", columns={\"date\": [\"day\", \"date\"], ...})\n", "# for note in notes: print(note)\n", "#\n", "# `columns` maps the names this notebook uses to the names your file might use (case and spacing do\n", "# not matter, and a currency suffix such as \"(EUR)\" is ignored). `date_columns` are read as dates,\n", "# `id_columns` are kept as text. Everything else that reads as numbers becomes numbers; a share\n", "# written \"49.41%\" becomes 0.4941. A share the platform shows only as \"< 10%\" or \"> 90%\" is read as\n", "# 0.10 or 0.90 and the direction is kept in a column ending \"_bound\", because it is a limit, not a\n", "# measurement. Every row set aside, and why, comes back in `notes`.\n", "import csv as _csv\n", "import io as _io\n", "import re as _re\n", "from pathlib import Path as _Path\n", "\n", "import numpy as _np\n", "import pandas as _pd\n", "\n", "\n", "class ExportReadError(ValueError):\n", " \"\"\"The file cannot be read without guessing; the message says what to change.\"\"\"\n", "\n", "\n", "_TOTAL_ROW = _re.compile(r\"^\\s*(grand\\s+)?totals?\\b\", _re.I)\n", "_CURRENCY = _re.compile(r\"[€$£¥₹]|\\b(EUR|USD|GBP|CHF|SEK|NOK|DKK|PLN|AUD|CAD)\\b\", _re.I)\n", "_EMPTY = {\"\", \"--\", \"-\", \"—\", \"–\", \"n/a\", \"na\", \"nan\", \"none\", \"null\", \"(not set)\"}\n", "_MONTHS = {m: i for i, m in enumerate(\n", " [\"jan\", \"feb\", \"mar\", \"apr\", \"may\", \"jun\", \"jul\", \"aug\", \"sep\", \"oct\", \"nov\", \"dec\"], 1)}\n", "\n", "\n", "def _norm(name):\n", " name = _re.sub(r\"\\((?:[A-Z]{3}|[€$£])\\)\", \"\", str(name))\n", " return _re.sub(r\"[\\s_\\-./]+\", \" \", name).strip().lower()\n", "\n", "\n", "def _decode(raw):\n", " if raw[:2] in (b\"\\xff\\xfe\", b\"\\xfe\\xff\"):\n", " return raw.decode(\"utf-16\")\n", " for encoding in (\"utf-8-sig\", \"cp1252\", \"latin-1\"):\n", " try:\n", " return raw.decode(encoding)\n", " except UnicodeDecodeError:\n", " continue\n", " return raw.decode(\"utf-8\", errors=\"replace\")\n", "\n", "\n", "def _rows_from(source):\n", " \"\"\"Every row of the file as a list of strings, whatever the container.\"\"\"\n", " if isinstance(source, _pd.DataFrame):\n", " return [list(map(str, source.columns))] + source.astype(str).values.tolist(), \",\"\n", " path = _Path(source)\n", " if not path.exists():\n", " raise ExportReadError(f\"There is no file at {path}. Check the name and the folder.\")\n", " if path.suffix.lower() in (\".xlsx\", \".xlsm\", \".xls\"):\n", " sheet = _pd.read_excel(path, header=None, dtype=str).fillna(\"\")\n", " return sheet.values.tolist(), \",\"\n", " text = _decode(path.read_bytes())\n", " lines = [ln for ln in text.splitlines() if not ln.lstrip().startswith(\"#\")]\n", " counts = {d: sum(ln.count(d) for ln in lines[:60]) for d in (\",\", \";\", \"\\t\")}\n", " delimiter = max(counts, key=counts.get) if any(counts.values()) else \",\"\n", " return [row for row in _csv.reader(_io.StringIO(\"\\n\".join(lines)), delimiter=delimiter)], delimiter\n", "\n", "\n", "def _find_header(rows, wanted):\n", " \"\"\"The first row that names the columns: the one matching most expected names, else the first\n", " row as wide as the data below it that is mostly words rather than numbers.\"\"\"\n", " widths = [len([c for c in r if str(c).strip()]) for r in rows]\n", " if wanted:\n", " best, best_hits = None, 0\n", " for i, row in enumerate(rows[:40]):\n", " names = {_norm(c) for c in row if str(c).strip()}\n", " hits = sum(1 for options in wanted.values() if names & set(options))\n", " if hits > best_hits:\n", " best, best_hits = i, hits\n", " if best is not None:\n", " return best\n", " body = [w for w in widths if w > 1]\n", " width = max(set(body), key=body.count) if body else 1\n", " for i, row in enumerate(rows[:40]):\n", " cells = [str(c).strip() for c in row if str(c).strip()]\n", " if len(cells) >= max(2, width - 1) and sum(_looks_number(c) for c in cells) <= len(cells) / 2:\n", " return i\n", " return 0\n", "\n", "\n", "def _looks_number(text):\n", " if _re.fullmatch(r\"\\d{1,4}[./\\-]\\d{1,2}[./\\-]\\d{1,4}\", str(text).strip()):\n", " return False # a date such as 22.09.2026, not a number with separators\n", " return bool(_re.fullmatch(r\"[<>]?\\s*[-+(]?[€$£]?\\s*[\\d.,\\s']+\\s*%?\\)?\", str(text).strip()))\n", "\n", "\n", "def _decimal_comma(values):\n", " \"\"\"True when these values write 1.234,56 (a comma followed by one or two final digits, or dots\n", " grouping thousands with a comma after them), False when they write 1,234.56, None when they do\n", " not say (whole numbers only, or a lone '1.340').\"\"\"\n", " comma_votes = dot_votes = 0\n", " for v in values:\n", " s = _re.sub(r\"[^\\d.,]\", \"\", v)\n", " if \",\" in s and \".\" in s:\n", " comma_votes += s.rfind(\",\") > s.rfind(\".\")\n", " dot_votes += s.rfind(\".\") > s.rfind(\",\")\n", " elif _re.search(r\",\\d{1,2}$\", s):\n", " comma_votes += 1\n", " elif _re.search(r\"\\.\\d{1,2}$\", s) or _re.search(r\"\\.\\d{4,}$\", s):\n", " dot_votes += 1\n", " if comma_votes == dot_votes:\n", " return None\n", " return comma_votes > dot_votes\n", "\n", "\n", "def _to_number(text, comma_decimal):\n", " s = str(text).strip().replace(\" \", \" \")\n", " if s.lower() in _EMPTY:\n", " return _np.nan, None, True\n", " bound = None\n", " if s[:1] in \"<>\":\n", " bound, s = s[0], s[1:].strip()\n", " negative = s.startswith(\"(\") and s.endswith(\")\")\n", " s = _CURRENCY.sub(\"\", s).strip(\"() \").replace(\" \", \"\").replace(\"'\", \"\")\n", " percent = s.endswith(\"%\")\n", " s = s.rstrip(\"%\")\n", " if comma_decimal:\n", " s = s.replace(\".\", \"\").replace(\",\", \".\")\n", " else:\n", " s = s.replace(\",\", \"\")\n", " try:\n", " value = float(s)\n", " except ValueError:\n", " return _np.nan, None, False\n", " if negative:\n", " value = -value\n", " return (value / 100.0 if percent else value), bound, True\n", "\n", "\n", "def _parse_dates(values, column, dayfirst):\n", " text = [str(v).strip() for v in values]\n", " present = [t for t in text if t.lower() not in _EMPTY]\n", " if not present:\n", " return _pd.Series([_pd.NaT] * len(text)), []\n", " notes = []\n", " if all(_re.fullmatch(r\"\\d{8}\", t) for t in present):\n", " return _pd.to_datetime(_pd.Series(text), format=\"%Y%m%d\", errors=\"coerce\"), notes\n", " if all(_re.fullmatch(r\"\\d{10}(\\.\\d+)?|\\d{13}|\\d{16}\", t) for t in present):\n", " digits = len(present[0].split(\".\")[0])\n", " unit = {10: \"s\", 13: \"ms\", 16: \"us\"}[digits]\n", " notes.append(f\"'{column}' holds epoch times in {dict(s='seconds', ms='milliseconds', us='microseconds')[unit]}; read as such.\")\n", " return _pd.to_datetime(_pd.to_numeric(_pd.Series(text), errors=\"coerce\"), unit=unit), notes\n", " cleaned = [_re.sub(r\"\\s+(UTC|GMT|Z)$\", \"\", t) for t in text]\n", " slash = [t for t in cleaned if _re.fullmatch(r\"\\d{1,2}[/.\\-]\\d{1,2}[/.\\-]\\d{2,4}(\\s.*)?\", t)]\n", " if slash and len(slash) >= len(present) * 0.9:\n", " firsts = [int(_re.split(r\"[/.\\-]\", t)[0]) for t in slash]\n", " seconds = [int(_re.split(r\"[/.\\-]\", t)[1]) for t in slash]\n", " dotted = all(\".\" in t.split(\" \")[0] for t in slash)\n", " if dayfirst is None:\n", " if max(firsts) > 12 and max(seconds) <= 12:\n", " dayfirst = True\n", " elif max(seconds) > 12 and max(firsts) <= 12:\n", " dayfirst = False\n", " elif dotted:\n", " dayfirst = True\n", " else:\n", " raise ExportReadError(\n", " f\"The dates in '{column}' (for example {slash[0]}) could be day-first or month-first, \"\n", " \"and nothing in the file says which. Re-run with dayfirst=True for 6 July written \"\n", " \"06/07, or dayfirst=False for June 7 written 06/07 - or export the dates as \"\n", " \"2026-07-06.\")\n", " return _pd.to_datetime(_pd.Series(cleaned), dayfirst=dayfirst, errors=\"coerce\"), notes\n", " parsed = _pd.to_datetime(_pd.Series(cleaned), errors=\"coerce\", format=\"mixed\")\n", " return parsed, notes\n", "\n", "\n", "def read_export(source, columns=None, date_columns=(), id_columns=(), dayfirst=None,\n", " max_unreadable_share=0.02):\n", " \"\"\"Read a downloaded report into a table with this notebook's column names. Returns\n", " (table, notes); raises ExportReadError with a plain-words fix when it cannot read without\n", " guessing.\"\"\"\n", " columns = columns or {}\n", " # Each name's candidates in the caller's order of preference, the name itself first.\n", " wanted = {name: list(dict.fromkeys([_norm(name)] + [_norm(a) for a in aliases]))\n", " for name, aliases in columns.items()}\n", " rows, delimiter = _rows_from(source)\n", " rows = [r for r in rows if any(str(c).strip() for c in r)]\n", " if not rows:\n", " raise ExportReadError(\"The file is empty.\")\n", " notes = []\n", " h = _find_header(rows, wanted)\n", " if h:\n", " notes.append(f\"set aside {h} line(s) above the header (a report title or date range).\")\n", " header = [str(c).strip() for c in rows[h]]\n", " body, totals = [], 0\n", " for r in rows[h + 1:]:\n", " first = next((str(c) for c in r if str(c).strip()), \"\")\n", " if _TOTAL_ROW.match(first):\n", " totals += 1\n", " continue\n", " if [str(c).strip() for c in r] == header:\n", " continue\n", " body.append([str(c) for c in r] + [\"\"] * (len(header) - len(r)))\n", " if totals:\n", " notes.append(f\"set aside {totals} total row(s).\")\n", " frame = _pd.DataFrame([r[:len(header)] for r in body], columns=header)\n", " rename = {}\n", " for name, options in wanted.items():\n", " by_norm = {}\n", " for c in frame.columns:\n", " by_norm.setdefault(_norm(c), c)\n", " match = next((by_norm[o] for o in options if o in by_norm and by_norm[o] not in rename), None)\n", " if match is not None:\n", " rename[match] = name\n", " frame = frame.rename(columns=rename)\n", " missing = [name for name in columns if name not in frame.columns]\n", " if missing:\n", " seen = \", \".join(f\"'{c}'\" for c in header[:12])\n", " raise ExportReadError(\n", " f\"Could not find the column(s) {', '.join(missing)}. The file's columns are {seen}. \"\n", " \"Rename the column in the file, or add its name to the list this cell accepts.\")\n", " dates = set(date_columns)\n", " # The file's own convention, for a column that cannot tell by itself: a clicks column of whole\n", " # numbers says nothing, but its \"1.340\" means one thousand three hundred and forty when the\n", " # file's other columns write decimals with a comma.\n", " every = [str(v) for c in frame.columns if c not in date_columns for v in frame[c].tolist()\n", " if _looks_number(v)]\n", " file_comma = _decimal_comma(every)\n", " if file_comma is None:\n", " file_comma = delimiter == \";\"\n", " ids = set(id_columns) | {c for c in frame.columns if _re.search(r\"(^|[ _])(id|pseudo id)$\", _norm(c))}\n", " for col in frame.columns:\n", " values = frame[col].tolist()\n", " if col in dates:\n", " parsed, extra = _parse_dates(values, col, dayfirst)\n", " notes += extra\n", " bad = int(parsed.isna().sum() - sum(str(v).strip().lower() in _EMPTY for v in values))\n", " if bad > max(1, max_unreadable_share * len(values)):\n", " example = next(v for v, p in zip(values, parsed) if _pd.isna(p) and str(v).strip())\n", " raise ExportReadError(f\"{bad} value(s) in '{col}' are not dates I can read, for example '{example}'.\")\n", " frame[col] = parsed.values\n", " continue\n", " if col in ids:\n", " continue\n", " present = [v for v in values if str(v).strip().lower() not in _EMPTY]\n", " if not present or sum(_looks_number(v) for v in present) < 0.5 * len(present):\n", " continue\n", " comma = _decimal_comma(present)\n", " if comma is None:\n", " comma = file_comma\n", " parsed = [_to_number(v, comma) for v in values]\n", " unreadable = [v for v, (_, _, ok) in zip(values, parsed) if not ok]\n", " if unreadable and col not in columns:\n", " notes.append(f\"'{col}' mixes numbers with labels such as '{unreadable[0]}'; kept as text.\")\n", " continue\n", " if unreadable:\n", " notes.append(f\"{len(unreadable)} value(s) in '{col}' could not be read as numbers, for example \"\n", " f\"'{unreadable[0]}'; they are left empty.\")\n", " if len(unreadable) > max(1, max_unreadable_share * len(values)):\n", " raise ExportReadError(\n", " f\"{len(unreadable)} of {len(values)} values in '{col}' are not numbers I can read, for \"\n", " f\"example '{unreadable[0]}'. Fix them in the file, or export the column unformatted.\")\n", " frame[col] = [p[0] for p in parsed]\n", " bounds = [p[1] for p in parsed]\n", " if any(bounds):\n", " # An object column, so a day with no limit stays None: a newer pandas would infer a string\n", " # column here and turn None into NaN, which is truthy and reads as a limit.\n", " frame[f\"{col}_bound\"] = _pd.Series(bounds, index=frame.index, dtype=object)\n", " notes.append(f\"'{col}' has values the platform shows only as a limit (for example \"\n", " f\"'{next(v for v, b in zip(values, bounds) if b)}'); kept as the limit, with the \"\n", " f\"direction in '{col}_bound'.\")\n", " if dates:\n", " undated = frame[list(dates)].isna().all(axis=1)\n", " if undated.any():\n", " frame = frame[~undated].reset_index(drop=True)\n", " notes.append(f\"set aside {int(undated.sum())} row(s) with no date (GA4 writes its grand total \"\n", " \"this way, under the header).\")\n", " return frame, notes\n", "# ---- end of export reader -------------------------------------------------------------------------\n", "\n", "\n", "COLUMN_ALIASES = {\n", " \"date\": [\"date\", \"day\", \"segments.date\", \"Day\", \"Date\"],\n", " \"series\": [\"series\", \"campaign\", \"campaign_name\", \"Campaign\", \"channel\",\n", " \"campaign.name\", \"ad_group\", \"Ad group\"],\n", " \"value\": [\"value\", \"metric\", \"cpa\", \"CPA\", \"cost_per_conversion\",\n", " \"Cost / conv.\", \"ctr\", \"CTR\"],\n", "}\n", "\n", "\n", "def find_column(df, key):\n", " \"\"\"Match a column by any of its known names, case- and space-insensitively.\"\"\"\n", " norm = {str(c).strip().lower().replace(\" \", \"\").replace(\"/\", \"\").replace(\".\", \"\"): c\n", " for c in df.columns}\n", " for alias in COLUMN_ALIASES[key]:\n", " k = alias.strip().lower().replace(\" \", \"\").replace(\"/\", \"\").replace(\".\", \"\")\n", " if k in norm:\n", " return norm[k]\n", " return None\n", "\n", "\n", "def load_my_data(path, metric=MY_METRIC, cost_col=COST_COL, conv_col=CONV_COL):\n", " \"\"\"Read a daily export into date / series / value, with plain-words errors.\n", "\n", " A day with no conversion has no cost per conversion: it is kept as an undefined day, never\n", " read as a cost of zero and never dropped as if it were missing.\"\"\"\n", " try:\n", " df, notes = read_export(path, columns={\"date\": COLUMN_ALIASES[\"date\"],\n", " \"series\": COLUMN_ALIASES[\"series\"]},\n", " date_columns=[\"date\"])\n", " except ExportReadError as problem:\n", " raise ValueError(str(problem)) from None\n", " for note in notes:\n", " print(\" \" + note)\n", " if cost_col and conv_col:\n", " lowered = {str(c).strip().lower(): c for c in df.columns}\n", " cost_name, conv_name = lowered.get(cost_col.strip().lower()), lowered.get(conv_col.strip().lower())\n", " if cost_name is None or conv_name is None:\n", " raise ValueError(\n", " f\"I could not find both '{cost_col}' and '{conv_col}' in the file. \"\n", " f\"The columns it has are: {', '.join(map(str, df.columns))}\"\n", " )\n", " cost = pd.to_numeric(df[cost_name], errors=\"coerce\")\n", " conv = pd.to_numeric(df[conv_name], errors=\"coerce\")\n", " value = cost.where(conv > 0) / conv.where(conv > 0)\n", " undefined = conv.notna() & (conv <= 0)\n", " else:\n", " value_col = find_column(df, \"value\") or metric\n", " if value_col not in df.columns:\n", " raise ValueError(\n", " f\"I could not find a column holding the number to watch. Rename it to \"\n", " f\"'value', or set COST_COL and CONV_COL. The columns it has are: \"\n", " f\"{', '.join(map(str, df.columns))}\"\n", " )\n", " value = pd.to_numeric(df[value_col], errors=\"coerce\")\n", " # a cost per conversion of 0.00 is how Google Ads writes a day with no conversion\n", " undefined = (value <= 0) if \"conv\" in str(value_col).lower() or \"cpa\" in str(value_col).lower() \\\n", " else pd.Series(False, index=df.index)\n", " value = value.where(~undefined)\n", " out = pd.DataFrame({\"date\": df[\"date\"], \"series\": df[\"series\"].astype(str), \"value\": value,\n", " \"undefined\": undefined.to_numpy()})\n", " out = out.dropna(subset=[\"date\"])\n", " out = out[out[\"value\"].notna() | out[\"undefined\"]]\n", " if out.empty:\n", " raise ValueError(\n", " \"Every entry was dropped - the date column did not parse as dates, or the \"\n", " \"value column held no numbers. Open the file and check those two columns.\"\n", " )\n", " return out.sort_values([\"series\", \"date\"]).reset_index(drop=True)\n", "\n", "\n", "def consolidate_touch_log(path, date_col=\"timestamp\", series_col=\"campaign\",\n", " value_col=None):\n", " \"\"\"Roll a raw per-event log up into the daily series this notebook wants.\"\"\"\n", " raw = pd.read_csv(path)\n", " raw[date_col] = pd.to_datetime(raw[date_col], errors=\"coerce\")\n", " raw = raw.dropna(subset=[date_col])\n", " grouped = raw.groupby([raw[date_col].dt.date, raw[series_col]])\n", " daily = grouped.size().rename(\"value\") if value_col is None else grouped[value_col].sum()\n", " out = daily.reset_index()\n", " out.columns = [\"date\", \"series\", \"value\"]\n", " out[\"date\"] = pd.to_datetime(out[\"date\"])\n", " return out\n", "\n", "\n", "print(\"loader defined\" + (\"\" if MY_FILE else \" - set MY_FILE above to use it\"))" ] }, { "cell_type": "markdown", "id": "93c7d5c3", "metadata": {}, "source": [ "## The checks that run before any number is printed\n", "\n", "The detectors above all need enough history to have something to compare against, and\n", "they need the days to be days. When they do not have that, the honest output is a\n", "refusal, not a smaller number.\n", "\n", "Three things are checked, and each one refuses in plain words:\n", "\n", "- **Enough days.** Fewer than four weeks and there is no baseline at all. The slope score\n", " needs 88 days (four weeks for a slope, then 60 days of past slopes to score it against),\n", " and the step search 90 settled days - 104 in all once the fortnight still filling in is\n", " set aside. Below those, each stays silent while the one-day score still runs.\n", "- **Gaps in the calendar.** Missing days are not zero days. A metric that skips weekends\n", " has a weekday rhythm the detector will read as a fortnightly cycle.\n", "- **A flat series.** If more than half the days hold the same value, the spread estimate\n", " is zero and every score is either zero or infinite. That is reported as\n", " \"no reading\", never as a number." ] }, { "cell_type": "code", "execution_count": 24, "id": "3ecc87b5", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:24:39.114222Z", "iopub.status.busy": "2026-10-25T13:24:39.113763Z", "iopub.status.idle": "2026-10-25T13:24:39.197012Z", "shell.execute_reply": "2026-10-25T13:24:39.195796Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "MY_FILE is not set - showing the refusals on deliberately broken data instead.\n", "\n", " ! 'Has a gap': 20 day(s) missing inside the date range. Missing days are not zero days - fill them or the weekday pattern will be read wrong.\n", " ! 'Has a gap': 100 days. The step search reads 90 settled days, so it needs 104 once the 14 still filling in are set aside. Step search left blank.\n", " ! 'Perfectly flat': more than half the days hold the same value, so there is no spread to measure against. No reading.\n", " ! 'Too short': only 10 days of history. The comparison window is 28 days, so there is nothing to compare today against yet. Come back at 29 days.\n" ] } ], "source": [ "def check_series(df_one, name):\n", " \"\"\"Return (ok_to_score, list_of_warnings) for one series.\"\"\"\n", " warns = []\n", " undefined = int(df_one[\"undefined\"].sum()) if \"undefined\" in df_one else 0\n", " if undefined > 0.1 * len(df_one):\n", " return False, [f\"'{name}': {undefined} of {len(df_one)} days had no conversion, so their cost per \"\n", " \"conversion does not exist. At this volume a daily cost per conversion is noise, not a \"\n", " \"signal: watch cost per click, conversions per week, or this campaign's weekly cost per \"\n", " \"conversion instead. No reading.\"]\n", " if undefined:\n", " warns.append(f\"'{name}': {undefined} day(s) had no conversion; they are left out of the scores, \"\n", " \"not counted as missing.\")\n", " df_one = df_one[~df_one[\"undefined\"]]\n", " n = len(df_one)\n", " if n < WINDOW + 1:\n", " return False, [f\"'{name}': only {n} days of history. The comparison window is \"\n", " f\"{WINDOW} days, so there is nothing to compare today against yet. \"\n", " f\"Come back at {WINDOW + 1} days.\"]\n", " span = (df_one[\"date\"].max() - df_one[\"date\"].min()).days + 1\n", " if span > n + undefined:\n", " warns.append(f\"'{name}': {span - n} day(s) missing inside the date range. Missing \"\n", " f\"days are not zero days - fill them or the weekday pattern will be \"\n", " f\"read wrong.\")\n", " v = df_one[\"value\"].to_numpy(dtype=float)\n", " if np.median(np.abs(v - np.median(v))) <= 0:\n", " return False, warns + [f\"'{name}': more than half the days hold the same value, so \"\n", " f\"there is no spread to measure against. No reading.\"]\n", " if n < WINDOW + 60:\n", " warns.append(f\"'{name}': {n} days is enough for the one-day detector but not for the \"\n", " f\"trend one, which needs {WINDOW + 60} days to know what a normal slope \"\n", " f\"looks like. Trend left blank.\")\n", " if n < 90 + SETTLE_DAYS:\n", " warns.append(f\"'{name}': {n} days. The step search reads 90 settled days, so it needs \"\n", " f\"{90 + SETTLE_DAYS} once the {SETTLE_DAYS} still filling in are set aside. \"\n", " f\"Step search left blank.\")\n", " return True, warns\n", "\n", "\n", "def score_my_data(df, cut_robust=None, cut_trend=None, settle_days=SETTLE_DAYS):\n", " \"\"\"Run all three detectors over every series in the reader's file.\n", "\n", " settle_days is the number of trailing days whose counts have not finished arriving.\n", " They are NOT scored by the day test. On a conversion metric this is the difference\n", " between a detector and a calendar: the measurement above fires on a large share of\n", " tracked metrics where nothing whatever happened, purely because the newest days are the\n", " emptiest and read adverse every time.\n", " \"\"\"\n", " results = []\n", " for name, block in df.groupby(\"series\"):\n", " block = block.sort_values(\"date\").reset_index(drop=True)\n", " ok, warns = check_series(block, name)\n", " if \"undefined\" in block:\n", " block = block[~block[\"undefined\"]].reset_index(drop=True)\n", " for w in warns:\n", " print(\" ! \" + w)\n", " if not ok:\n", " continue\n", " v = block[\"value\"].to_numpy(dtype=float)\n", " d = block[\"date\"].dt.dayofweek.to_numpy()\n", " rz = robust_scores(v, d)\n", " tz = trend_scores(v, d) if len(v) >= WINDOW + 60 else np.full(len(v), np.nan)\n", " cr = CUT_ROBUST if cut_robust is None else cut_robust\n", " ct = CUT_TREND if cut_trend is None else cut_trend\n", "\n", " # The newest day the day test is allowed to score. Everything after it is still\n", " # filling in, so a score there measures the reporting curve, not the account.\n", " scored = len(v) - 1 - int(settle_days)\n", " if scored < WINDOW:\n", " print(f\" ! '{name}': after setting aside the {settle_days} day(s) still \"\n", " f\"filling in, there is not enough settled history to score. No reading.\")\n", " continue\n", " # The step search reads the settled days only: the unsettled fortnight reads low on a\n", " # conversion metric and would show as a step down that never happened.\n", " bd = break_days(v[:scored + 1], d[:scored + 1], penalty_sd=PENALTY)\n", " if settle_days > 0:\n", " recent = np.abs(rz[scored + 1:])\n", " unusable = int(np.nansum(recent > cr))\n", " if unusable:\n", " print(f\" ! '{name}': {unusable} of the {settle_days} unsettled day(s) \"\n", " f\"would have fired. Not reported - those counts are incomplete.\")\n", "\n", " results.append({\n", " \"series\": name,\n", " \"days\": len(v),\n", " \"scored_day\": str(block[\"date\"].iloc[scored].date()),\n", " \"days_set_aside\": int(settle_days),\n", " \"blip_score\": None if np.isnan(rz[scored]) else round(float(rz[scored]), 2),\n", " \"blip_fired_recently\": bool(\n", " np.nansum(np.abs(rz[max(0, scored - 13):scored + 1]) > cr) > 0),\n", " \"trend_score\": None if np.isnan(tz[scored]) else round(float(tz[scored]), 2),\n", " \"trend_fired\": bool(not np.isnan(tz[scored]) and abs(tz[scored]) > ct),\n", " \"break_detected_on\": [str(block[\"date\"].iloc[c].date()) for c in bd],\n", " })\n", " return pd.DataFrame(results)\n", "\n", "\n", "if MY_FILE:\n", " my_df = load_my_data(MY_FILE)\n", " print(f\"loaded {len(my_df)} entries across {my_df['series'].nunique()} tracked metrics\\n\")\n", " my_results = score_my_data(my_df)\n", " print(my_results.to_string(index=False))\n", " print(\"\\nRemember the arithmetic above: across \"\n", " f\"{my_df['series'].nunique()} series you should expect some of these to be \"\n", " \"false alarms. Read the alarm-budget table before acting on any single one.\")\n", "else:\n", " print(\"MY_FILE is not set - showing the refusals on deliberately broken data instead.\\n\")\n", " short = pd.DataFrame({\n", " \"date\": pd.date_range(\"2026-01-01\", periods=10),\n", " \"series\": \"Too short\",\n", " \"value\": np.linspace(20, 22, 10),\n", " })\n", " flat = pd.DataFrame({\n", " \"date\": pd.date_range(\"2026-01-01\", periods=90),\n", " \"series\": \"Perfectly flat\",\n", " \"value\": 20.0,\n", " })\n", " gappy = pd.DataFrame({\n", " \"date\": pd.to_datetime(pd.date_range(\"2026-01-01\", periods=120)[\n", " np.r_[0:40, 60:120]]),\n", " \"series\": \"Has a gap\",\n", " \"value\": rng.normal(20, 1.5, 100),\n", " })\n", " _ = score_my_data(pd.concat([short, flat, gappy], ignore_index=True))" ] }, { "cell_type": "markdown", "id": "9075b256", "metadata": {}, "source": [ "## What this notebook will not tell you\n", "\n", "Three limits worth carrying away, because they are the ones that bite.\n", "\n", "**It cannot tell you why.** Every detector here answers \"did the level, the direction or\n", "the shape of this series change\" and none of them answers \"because of what\". A break on\n", "the day you changed your bidding strategy and a break on the day a competitor raised\n", "theirs look identical. Cross-referencing detected boundaries against your own change log\n", "is the cheapest thing you can do to close that gap, and it is a lookup, not a model.\n", "\n", "**It assumes the series jumps rather than drifts.** The break detector looks for a step\n", "to a new level. A metric that is sliding smoothly - gradual audience fatigue, a slowly\n", "saturating market - will either be reported as a step that never happened or missed\n", "entirely, and neither is a useful answer. When the picture is a slope rather than a\n", "staircase, this family of methods is the wrong one, and a trend-following or forecasting\n", "model is the right one.\n", "\n", "**A very recent break is not yet a break.** A boundary detected three days ago has three\n", "days of evidence on one side of it. It may firm up into a real shift or it may evaporate.\n", "Treat the most recent stretch as provisional and let the next few days settle it." ] }, { "cell_type": "code", "execution_count": 25, "id": "7f2fc402", "metadata": { "execution": { "iopub.execute_input": "2026-10-25T13:24:39.202198Z", "iopub.status.busy": "2026-10-25T13:24:39.201784Z", "iopub.status.idle": "2026-10-25T13:24:39.208942Z", "shell.execute_reply": "2026-10-25T13:24:39.208098Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Done. Synthetic throughout; every figure and every number above regenerates from seed 20260910.\n" ] } ], "source": [ "print(\"Done. Synthetic throughout; every figure and every number above regenerates from \"\n", " f\"seed {SEED}.\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 5 }