Skip to content
ML to LLMs
Labs / AI Engineering

A/B test power and peeking

Compute sample size from baseline, effect, and power, then simulate how peeking inflates false positives.

Theory and worked examples: Experimentation and A/B testing

A/B test power and peeking

Top: how many users a conversion-rate test needs. Bottom: 2,000 seeded A/A tests (no real difference) showing what checking the p-value every day does to your false-positive rate.

Power versus true lift051015202500.20.40.60.81true relative lift (%)power
power with n = 14,749 per armMDEtarget power
False-positive rate when peeking daily51000.10.20.3number of daily looksfalse-positive rate
stop at first significant looknominal α = 0.05

To detect a 10% relative lift on a 10.0% baseline with α = 0.05 and power 0.80, you need 14,749 users per arm: 10 days at 1,500 per arm per day. Halving the MDE roughly quadruples n.

In 2,000 A/A tests, one look at the end flags 4.9% as significant (close to α). Stopping at the first daily p < 0.05 over 14 days flags 22.1%: every extra look is another chance for noise to cross the line.