Why correct for multiple tests
Each test at α = 0.05 has a 5% chance of a false positive when there is no real effect. Run many tests and false positives become almost certain: with 10 independent tests, the probability of at least one false positive, the family-wise error rate, is 1 − 0.95¹⁰ ≈ 40%. This is how studies that test many outcomes, subgroups or genes produce spurious “discoveries”.
The corrections
- Bonferroni: test each hypothesis at α ÷ m, where m is the number of tests. Equivalently, multiply each p-value by m (capped at 1) and compare it with α.
- Šidák: test each hypothesis at 1 − (1 − α)^(1/m). It is exact for independent tests and very slightly less strict than Bonferroni.
Both keep the family-wise error rate at or below α.
Example
You run 10 tests at α = 0.05, and one gives p = 0.004:
- Bonferroni threshold: 0.05 ÷ 10 = 0.005; Šidák threshold: 0.00512;
- without correction, the family-wise error rate is 40.1%, and you would expect 0.5 false positives if no effect were real;
- the adjusted p-value is 0.004 × 10 = 0.04, below 0.05: the result remains significant after correction.
A p-value of 0.02, by contrast, would be significant alone but not after correction (adjusted p = 0.2).
Holm’s method: a more powerful alternative
Holm’s step-down procedure also controls the family-wise error rate but rejects more often:
- Sort the p-values from smallest to largest.
- Compare the smallest with α ÷ m, the next with α ÷ (m − 1), and so on.
- Stop at the first p-value that fails its threshold; it and all larger ones are not significant.
There is no reason to prefer plain Bonferroni over Holm, except simplicity.
When to correct, and when not to
- Correct when a single significant result among several tests would lead to a claim, such as “the drug improves at least one outcome”.
- Pre-specified primary outcomes are often tested without correction, while secondary outcomes are corrected or labeled exploratory.
- With hundreds or thousands of tests, as in genomics, controlling the false discovery rate (Benjamini–Hochberg) is usually more useful than the very strict Bonferroni threshold.
The cost of the correction is lower power: the more tests you run, the larger an effect must be to be detected.