The Krippendorff’s alpha calculator for two coders gives the agreement beyond chance between two coders who marked units yes or no, from the number of units they both marked yes, both marked no and those on which they disagreed, with the percent agreement.
How the calculation works
- Enter the number of units both coders marked yes and both marked no.
- Enter the number of units on which they disagreed.
- Read alpha, the percent agreement and the disagreements.
Example
With 40 units marked yes by both, 30 marked no by both and 10 disagreements, there are 80 units. The percent agreement is 87.5 %, but the expected disagreement is 0.4953 against an observed 0.1250, which gives an alpha of 0.748. The code yes is used in 56.3 % of all codes.
Why it matters
The calculator helps researchers in content analysis, qualitative coding, medical annotation and machine learning labeling who must report the reliability of a binary coding by two people, and compare it with the usual thresholds for publication.
Good to know
Count each unit once, with both codes known. For more than two coders, missing codes or more than two categories, use the general formula of alpha with a coincidence matrix. Alpha depends on the prevalence of the code: when one code is rare, a few disagreements lower alpha a lot. Report both the percent agreement and alpha, together with the number of units, so that readers can judge the result. Train the coders on a sample before the real coding, write a clear coding manual, and check the agreement again during the work to detect any drift of the coders over time.
Related tools
The Cohen kappa calculator gives a similar index for two raters, the intraclass correlation calculator handles continuous ratings, the Lin concordance correlation calculator checks agreement of measurements and the confusion matrix calculator summarizes classifications.