Why sample size matters in A/B testing
An A/B test compares a control page (A) with a variant (B). Stop it too early and you risk a false winner; run it too small and a real improvement goes unnoticed. Deciding the sample size before the test fixes both problems and tells you how long the test must run.
The inputs
- Baseline conversion rate: the current rate of the control, such as 5% of visitors who buy.
- Minimum detectable effect (MDE): the smallest relative lift worth detecting, such as +20% (from 5% to 6%).
- Significance level α: usually 5%, the risk of declaring a winner that is not one.
- Power: usually 80%, the chance of detecting the effect if it is real.
The formula
For two proportions p₁ and p₂, with p̄ = (p₁ + p₂) ÷ 2, each group needs:
n = [z₁₋α/₂ √(2p̄(1 − p̄)) + z₁₋β √(p₁(1 − p₁) + p₂(1 − p₂))]² ÷ (p₂ − p₁)²
Example
Baseline 5%, MDE +20% relative, α = 5% two-tailed, 80% power, 2,000 visitors a day split between two variants:
- target rate: 6%, an absolute difference of 1 point;
- 8,158 visitors per variant, 16,316 in total;
- at 2,000 visitors a day: 9 days. Round up to 14 days to cover two full weeks.
What drives sample size
- Smaller effects need many more visitors: halving the MDE roughly quadruples the sample. Detecting +10% instead of +20% here would take about 31,000 visitors per variant.
- Lower baseline rates need larger samples for the same relative lift.
- More variants multiply the total traffic needed, and call for a correction such as Bonferroni.
Good practice
- Do not peek and stop as soon as p < 0.05: repeated checking inflates false positives. Wait for the planned sample, or use a sequential testing method designed for early stopping.
- Run tests over whole weeks, since behavior differs between weekdays and weekends.
- Choose the MDE from business value: what lift would justify shipping the change?
- Check that the traffic split is as planned (a sample ratio mismatch suggests a bug).
When the test ends, the A/B test significance calculator gives the p-value and confidence interval of the observed difference.