What a p-value is
In a statistical test, you start from a null hypothesis, usually “there is no effect” or “there is no difference”. The p-value is the probability of getting a result at least as extreme as the one observed if that null hypothesis were true. A small p-value means the data would be surprising under the null hypothesis, which counts as evidence against it. This calculator converts a z test statistic into a p-value and compares it with your significance level α.
The formulas
With Φ the standard normal cumulative distribution:
- Two-tailed: p = 2 × (1 − Φ(|z|))
- Right-tailed (testing for an increase): p = 1 − Φ(z)
- Left-tailed (testing for a decrease): p = Φ(z)
- Critical value for a two-tailed test: z* = Φ⁻¹(1 − α/2)
The result is significant when p < α, which is the same as |z| > z*.
Worked example
A test of a new teaching method gives a z statistic of 2.1. You planned a two-tailed test at α = 0.05.
- Φ(2.1) = 0.98214.
- p = 2 × (1 − 0.98214) = 0.0357.
- 0.0357 < 0.05, so the result is statistically significant: you reject the null hypothesis of no effect.
- Equivalently, |z| = 2.1 exceeds the critical value 1.96.
- At the stricter α = 0.01, the critical value is 2.576 and the result would not be significant.
Z statistics and p-values
| z | Two-tailed p | One-tailed p |
|---|---|---|
| 1.000 | 0.3173 | 0.1587 |
| 1.645 | 0.1000 | 0.0500 |
| 1.960 | 0.0500 | 0.0250 |
| 2.326 | 0.0200 | 0.0100 |
| 2.576 | 0.0100 | 0.0050 |
| 3.000 | 0.0027 | 0.0013 |
| 3.291 | 0.0010 | 0.0005 |
Choosing one or two tails
Use a two-tailed test unless you decided before collecting data that only one direction matters and that an effect in the other direction would be treated exactly like no effect. Switching to a one-tailed test after seeing the data halves the p-value and is a well-known way to manufacture significance.
What a p-value is not
The American Statistical Association’s 2016 statement on p-values stresses a few points worth remembering:
- A p-value is not the probability that the null hypothesis is true, nor the probability that the result is due to chance.
- p < 0.05 is a convention, not a law of nature. A p-value of 0.049 and one of 0.051 carry almost the same evidence.
- Statistical significance is not practical importance. With a large enough sample, a trivial difference gives a tiny p-value. Report the effect size and its confidence interval.
- Running many tests and reporting only the significant ones (p-hacking) produces false discoveries. Adjust for multiple comparisons.
For t tests with small samples, chi-square tests or F tests, use the matching distribution; this calculator covers z tests and large-sample approximations. For a complete comparison of two conversion rates, use the A/B test calculator.