Is your A/B test underpowered? A guide to statistical power
By Mo Touzani · · Experimentation
Statistical power is the chance your test detects a real effect when one exists. Few teams calculate it before launch. They split traffic, check the dashboard each morning and ship the variant that looks ahead. That habit produces a steady stream of winners that never show up in revenue.
How peeking breaks a test
A 95% significance threshold assumes you look at the result once, at a sample size you fixed in advance. Each extra look gives random noise another chance to cross the line. Check a test ten times and stop at the first p-value under 0.05, and your false-positive rate climbs from 5% to about 19%. Check it twenty times and you reach about 25%.
The p-value your tool displays assumes a single look. It does not know you looked eleven times. Johari and colleagues at Stanford and Optimizely documented this effect on live experimentation platforms in 2017, and it remains the most common way teams fool themselves.
Running a test until it turns significant is a method for producing false positives.
The three inputs that set your sample size
You need three numbers before launch: the significance level (95% is standard), the power you want (80% is standard) and the minimum detectable effect, or MDE, the smallest lift worth acting on.
The MDE drives everything. At a 2% conversion rate, detecting a 20% relative lift (2.0% to 2.4%) at 95% significance and 80% power takes about 21,000 visitors per variant. Detecting a 5% lift (2.0% to 2.1%) takes about 315,000. Cutting the lift you want to see by four multiplies the traffic by fifteen.
At 80% power you still miss a real effect one time in five. A test that ends without significance proves nothing on its own; the variant may work and your sample may be too small to show it.
Three ways tests fail
- Underpowered. The test runs on too little traffic for the lift you expected. If you peek until it turns significant, you get a false positive. If you wait, you get a confidence interval wide enough to include a big win and a big loss, which tells you nothing.
- Peeked. You stop the test the first time it looks significant. The stated 5% error rate no longer applies, and the real rate can be four or five times higher.
- Too many comparisons. You test several variants or several metrics against one threshold. Ten variants at p < 0.05 produce about one false winner in every two tests, even if none of them work. Bonferroni or Benjamini-Hochberg corrections fix this, and few teams apply them.
How to pick the minimum detectable effect
Ask what lift would change your roadmap. Say your checkout converts at 3.2%. A 5% relative lift takes it to 3.36% and needs about 195,000 visitors per variant. If that lift would justify the engineering work, size for it. If you would only act on 15% or more, size for 15% and accept that smaller gains will slip through.
Match the MDE to the cost of the change. A headline or button change costs little, so a 5% to 10% MDE makes sense if you have the traffic. A checkout rebuild costs a lot, so you only care about larger effects, and those need less traffic to detect.
Treat the calculated sample as a floor. You cannot read the test before it arrives, however good the numbers look. Our free sample size calculator at sinfa.com/tools gives you the number and the days it will take at your traffic.
When you cannot wait
Sometimes a losing variant costs too much to leave running. Sequential methods let you monitor a test as it runs without inflating the error rate, as long as you choose them before launch. Optimizely's Stats Engine and VWO's SmartStats both work this way.
Bayesian versions answer a different question: how likely is each size of effect, given the data? Product and finance teams often find that easier to act on. The trade-off is a prior belief about the likely effect, which you have to state and defend.
What a null result tells you
Suppose a test sized for your MDE ends without significance. Look at the confidence interval. If it sits tight around zero, the variant performs about the same as control: log it, update your hypothesis and move on. If it spans large gains and large losses, the test was too small, and you should rerun it at the right size before you judge the idea.
A well-powered null result rules out any lift above your MDE. Teams that record these results build a map of what does not move conversion on their site. That map holds up longer than a list of winners that may not replicate.
Sources: Johari, R. et al., "Peeking at A/B Tests: Why It Matters, and What to Do About It," ACM SIGKDD (2017). Deng, A. et al., "Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data," ACM WSDM (2013). Kohavi, R., Tang, D. and Xu, Y., Trustworthy Online Controlled Experiments, Cambridge University Press (2020).