Skip to content
Omnikalc

Krippendorff’s alpha calculator (two coders, binary)

Enter the number of units both coders marked yes, both marked no, and those they disagreed on to get Krippendorff’s alpha, the observed and expected disagreement and the percent agreement.

Updated on
Your inputs

Value out of range

Value out of range

Value out of range

Results

Krippendorff’s alpha

0.748

Percent agreement

87.5 %

Units coded
80
Observed disagreement
0.1250
Expected disagreement
0.4953
Share of yes among all codes
56.3 %

The Krippendorff’s alpha calculator for two coders gives the agreement beyond chance between two coders who marked units yes or no, from the number of units they both marked yes, both marked no and those on which they disagreed, with the percent agreement.

How the calculation works

  • Enter the number of units both coders marked yes and both marked no.
  • Enter the number of units on which they disagreed.
  • Read alpha, the percent agreement and the disagreements.

Example

With 40 units marked yes by both, 30 marked no by both and 10 disagreements, there are 80 units. The percent agreement is 87.5 %, but the expected disagreement is 0.4953 against an observed 0.1250, which gives an alpha of 0.748. The code yes is used in 56.3 % of all codes.

Why it matters

The calculator helps researchers in content analysis, qualitative coding, medical annotation and machine learning labeling who must report the reliability of a binary coding by two people, and compare it with the usual thresholds for publication.

Good to know

Count each unit once, with both codes known. For more than two coders, missing codes or more than two categories, use the general formula of alpha with a coincidence matrix. Alpha depends on the prevalence of the code: when one code is rare, a few disagreements lower alpha a lot. Report both the percent agreement and alpha, together with the number of units, so that readers can judge the result. Train the coders on a sample before the real coding, write a clear coding manual, and check the agreement again during the work to detect any drift of the coders over time.

The Cohen kappa calculator gives a similar index for two raters, the intraclass correlation calculator handles continuous ratings, the Lin concordance correlation calculator checks agreement of measurements and the confusion matrix calculator summarizes classifications.

Frequently asked questions

What is Krippendorff’s alpha?

Krippendorff’s alpha is a measure of agreement between coders that corrects for the agreement expected by chance. It is defined as α = 1 − Do / De, where Do is the observed disagreement and De the disagreement expected if the codes were assigned at random with the same frequencies. A value of 1 is perfect agreement and 0 is chance level.

How does it differ from percent agreement?

Percent agreement counts only the share of units on which the coders agree. When one code is very common, the coders agree often just by chance and the percent agreement looks high. Alpha removes the chance part, so a data set with 90 percent agreement can have a low alpha if almost every unit is coded yes.

Which value is acceptable?

Krippendorff proposed that α ≥ 0.80 is reliable, and 0.667 ≤ α < 0.80 allows only tentative conclusions. Lower values mean that the data are not reliable enough for most purposes. The thresholds are guidelines and depend on the cost of errors in the use of the data.

Why use alpha instead of Cohen’s kappa?

Alpha handles any number of coders, missing data and any type of scale, while Cohen’s kappa is limited to two coders. For two coders and a binary code the values are very close, and alpha is slightly lower for small samples because it corrects for the sample size. This calculator covers the two-coder binary case.

Was this calculator helpful?

Thanks for your feedback!

Results are estimates provided for information purposes. They do not replace professional advice.