Most of your pages can’t carry an A/B test.

Every valid A/B test has a minimum size, settled by arithmetic before anybody has an opinion about the button. Here is the standard formula.

Power, in plain language, is the chance a test could have spotted the effect at all, supposing it was really there. The paper’s worked example opens on a 3.7% baseline conversion rate and a target of detecting a 10% relative change, which is 0.37 percentage points.

The test those authors were pulling apart had run on roughly 80 users per variant, which puts its power, before it ever launched, at about 3%. It was never capable of finding what it reported, and the same arithmetic sits underneath the tests being sold to you.

The piece worth carrying around is the denominator, because δ gets squared: halve the effect you are trying to detect and the traffic you need goes up fourfold, whatever the tool and whatever the hypothesis. The same formula, against more typical lead-gen baselines:

Their rule of thumb: A/B tests pick up effects of a reasonable size once you have thousands of active users in each variant, and tens of thousands is better. Hold that against the monthly sessions on one city page.

An underpowered test doesn’t simply fail to find things, it corrupts whatever it does find, and how badly depends on how often your ideas turn out to be true. A 10% idea-success rate is what Booking.com, Google Ads and Netflix report, so that is the rate the table assumes.

Watch a fixed-horizon test continuously, call it the moment the dashboard goes green, and you can inflate the false-positive rate fivefold, even at 10,000 samples. Across 2,101 commercial experiments on Optimizely, 57% of experimenters stopped early on a good-looking result, and around 73% halted exactly as a positive effect crossed 90% confidence. The damage is a false discovery rate climbing from 33% to 42%, on top of roughly 75% of tested effects being truly null to begin with.

This is the piece most CRO pitches leave out. Standard client-side tools change the page after it loads, so the H1 you think you are testing may never be the H1 Googlebot renders. That’s fine for a button below the fold and useless for copy that has to rank. Google’s testing documentation (last updated December 10, 2025) sets four rules:

Where the volume isn’t there, two alternatives exist and we use both. Page-group split testing randomizes pages rather than users. You split a template’s pages into similar control and variant sets and compare group against group. A set of a thousand location pages becomes testable even when no single page in it could be. SearchPilot’s published threshold is hundreds of pages on one template, 30,000+ organic sessions a month reaching the group under test, and tests running 2–4 weeks. A practitioner threshold rather than a law, and we still find it useful.

Counterfactual modelling, Google’s published CausalImpact method, fits a model over matched control pages and estimates what would have happened had you changed nothing. Weaker evidence than a randomized test, and we label it weaker evidence. We use it where the alternative is a before-and-after chart. Where a real test was there to run, we run the test.

The figures below are traffic figures, not conversion figures.