Why effect size matters
A p-value says whether a difference is likely to be real; it does not say whether it is large. With enough data, trivial differences become “significant”. Effect size measures how big the difference is, independent of sample size. Cohen’s d is the most common one for comparing two means.
The formulas
- Pooled standard deviation sp = √[((n₁ − 1)s₁² + (n₂ − 1)s₂²) ÷ (n₁ + n₂ − 2)]
- Cohen’s d = (m₁ − m₂) ÷ sp
- Hedges’ g = d × (1 − 3 ÷ (4(n₁ + n₂) − 9))
- Probability of superiority = Φ(d ÷ √2)
- Cohen’s U3 = Φ(d), the share of group 2 below the mean of group 1
Worked example
A training program: group 1 scores 105 on average, group 2 100, both with a standard deviation of 15 and 50 people each.
- Pooled SD: 15.
- d = (105 − 100) ÷ 15 ≈ 0.33, a small effect.
- Hedges’ g ≈ 0.331.
- Probability of superiority ≈ 59%: a random member of group 1 outscores a random member of group 2 59% of the time.
- U3 ≈ 63% of group 2 scores below group 1’s average.
Cohen’s benchmarks
| d | Effect | Overlap of the two distributions |
|---|---|---|
| 0.2 | small | about 92% |
| 0.5 | medium | about 80% |
| 0.8 | large | about 69% |
| 1.2 | very large | about 55% |
Cohen offered these as rough guides for when nothing better is known. In education, an effect of 0.2 to 0.4 from a teaching program is often considered meaningful; in medicine, the clinical importance of a difference matters more than any benchmark.
Uses
- Comparing studies: d puts results measured on different scales on a common footing, which is why meta-analyses use it.
- Planning sample sizes: detecting d = 0.2 with 80% power at the 5% level needs about 394 people per group; d = 0.5 needs about 64; d = 0.8 about 26.
- Reporting: many journals ask for effect sizes alongside p-values.
Cautions
- d assumes roughly normal data with similar standard deviations in both groups. With very unequal spreads, consider Glass’s delta, which uses the control group’s SD.
- For paired measurements, such as before and after on the same people, a different formula based on the standard deviation of the differences applies.