Did version B really win?
In an A/B test, you show two versions of a page, email or ad to similar groups of visitors and compare how often each one converts. Version B will almost always do a little better or a little worse just by chance. The question is whether the difference is large enough, given the number of visitors, to be confident that it is real. This calculator runs a two-proportion z-test and gives the conversion rates, the relative uplift, the p-value and a confidence interval for the difference.
The formulas
With n visitors and c conversions in each group:
- Conversion rates: p_A = c_A ÷ n_A and p_B = c_B ÷ n_B
- Pooled rate: p = (c_A + c_B) ÷ (n_A + n_B)
- Standard error: SE = √(p × (1 − p) × (1/n_A + 1/n_B))
- z = (p_B − p_A) ÷ SE, and the two-tailed p-value = 2 × (1 − Φ(|z|))
- Confidence interval of the difference: (p_B − p_A) ± z* × √(p_A(1 − p_A)/n_A + p_B(1 − p_B)/n_B)
Worked example
An online store tests a new checkout button. Each version gets 10,000 visitors: A converts 200 (2.00%), B converts 245 (2.45%).
- Relative uplift: 2.45 ÷ 2.00 − 1 = +22.5%.
- Pooled rate: 445 ÷ 20,000 = 2.225%; SE = 0.209 points.
- z = 0.45 ÷ 0.209 = 2.16, giving p = 0.031.
- At 95% confidence the difference is significant, and B’s true advantage is likely between +0.04 and +0.86 points: positive, but possibly much smaller than the observed 0.45.
How much difference is significant
Keeping A at 200 conversions out of 10,000:
| Conversions in B | Uplift | P-value |
|---|---|---|
| 210 | +5.0% | 0.62 |
| 220 | +10.0% | 0.32 |
| 230 | +15.0% | 0.14 |
| 245 | +22.5% | 0.031 |
| 260 | +30.0% | 0.005 |
| 280 | +40.0% | 0.0002 |
Even a 15% uplift is not distinguishable from chance with 10,000 visitors per version at a 2% conversion rate.
Good testing practice
- Fix the sample size in advance. Use the expected conversion rate and the smallest uplift worth detecting. Detecting +20% on a 2% rate takes about 21,000 visitors per variant at 95% confidence and 80% power.
- Do not peek and stop. Checking results every day and stopping at the first p < 0.05 can push the false positive rate above 20%.
- Run full weeks. Behavior differs between weekdays and weekends; run at least one or two complete weekly cycles.
- Test one main metric. Looking at many metrics or segments multiplies the chances of a fluke.
- Beware novelty effects. A new design can attract attention at first and lose its edge later.
A significant result tells you the difference is probably real, not that it is large. Check the confidence interval and the business value of the change before rolling it out. For the advertising side of the funnel, see the ROAS and CPC calculator.