What Cohen’s kappa measures
When two raters classify the same items, such as two doctors reading X-rays, two teachers grading essays as pass or fail, or two coders categorizing interviews, the simple percentage of agreement is misleading: some agreement happens by chance. Cohen’s kappa corrects for it and measures agreement beyond chance.
The formula
κ = (pₒ − pₑ) ÷ (1 − pₑ)
- pₒ, observed agreement: the share of items on the diagonal of the table, where both raters chose the same category;
- pₑ, chance agreement: Σ (row totalᵢ × column totalᵢ) ÷ N².
κ = 1 means perfect agreement, κ = 0 means agreement no better than chance, and a negative κ means systematic disagreement.
Example
Two raters classify 60 items into three categories. They agree on 20, 15 and 10 items, so pₒ = 45 ÷ 60 = 75%. Rater A’s totals are 25, 22 and 13; rater B’s are 25, 20 and 15.
- pₑ = (25 × 25 + 22 × 20 + 13 × 15) ÷ 3,600 = 35%;
- κ = (0.75 − 0.35) ÷ 0.65 ≈ 0.62;
- 95% confidence interval: about 0.45 to 0.78.
For two categories, enter zeros in the third row and column.
Interpreting kappa
The scale proposed by Landis and Koch (1977) is the most cited:
| κ | Agreement |
|---|---|
| below 0 | poor |
| 0 to 0.20 | slight |
| 0.21 to 0.40 | fair |
| 0.41 to 0.60 | moderate |
| 0.61 to 0.80 | substantial |
| 0.81 to 1 | almost perfect |
These thresholds are conventions. In clinical settings, many authors expect at least 0.6 to 0.7 before trusting a classification.
Pitfalls
- Prevalence effect: when one category is very common, pₑ is high and κ can be low even with 90% agreement. Report pₒ alongside κ.
- Ordered categories such as mild, moderate and severe: a weighted kappa, which gives partial credit for near misses, is more appropriate.
- More than two raters: use Fleiss’ kappa.
- The confidence interval uses a simple large-sample approximation of the standard error. With fewer than about 50 items, treat it as indicative.