Skip to content
Omnikalc

Intraclass correlation calculator: how reliable are your raters?

Enter the mean squares from a two-way ANOVA, with the numbers of subjects and raters, to get the five common intraclass correlation coefficients.

Updated on
Your inputs

Value out of range

Value out of range

Value out of range

Value out of range

Value out of range

Results

ICC(2,1): single rater, two-way random, absolute agreement

0.835

Good reliability
ICC(1,1): single rater, one-way random
0.835
ICC(3,1): single rater, two-way mixed, consistency
0.837
ICC(2,k): mean of k raters, absolute agreement
0.938
ICC(3,k): mean of k raters, consistency
0.939

The intraclass correlation coefficient, or ICC, measures the reliability of ratings or measurements made by several raters. The intraclass correlation calculator takes the mean squares from a two-way ANOVA, the number of subjects and the number of raters, and gives ICC(1,1), ICC(2,1), ICC(3,1), ICC(2,k) and ICC(3,k).

How the calculation works

  • Run a two-way analysis of variance on the ratings, with subjects as rows and raters as columns. Read the mean squares for subjects (MSR), raters (MSC) and error (MSE).
  • ICC(3,1) = (MSR − MSE) ÷ (MSR + (k − 1) × MSE) measures consistency for a single rater, where k is the number of raters.
  • ICC(2,1) = (MSR − MSE) ÷ (MSR + (k − 1) × MSE + k × (MSC − MSE) ÷ n) measures absolute agreement, and it is lower when raters differ systematically.
  • ICC(1,1) uses the within-subject mean square MSW = (MSC + (n − 1) × MSE) ÷ n. The versions for the mean of k raters are ICC(2,k) = (MSR − MSE) ÷ (MSR + (MSC − MSE) ÷ n) and ICC(3,k) = (MSR − MSE) ÷ MSR.

Example

Three raters score 20 subjects. The ANOVA gives MSR = 52.6, MSC = 4.1 and MSE = 3.2. Then ICC(3,1) = 49.4 ÷ 59.0 = 0.837, and ICC(2,1) = 49.4 ÷ 59.135 = 0.835. ICC(1,1) is 0.835, and the reliability of the mean of the three raters is 0.938 for absolute agreement and 0.939 for consistency. A single rater is in the good range.

Why it matters

Before using a scale, a test or a panel of raters, you need to know whether the scores can be trusted. The ICC answers that with a single number, and the five forms match the common study designs, so it can be reported in the way journals expect.

Good to know

The formulas follow Shrout and Fleiss and use mean squares from an ANOVA without missing data. They give point estimates and not confidence intervals, which need the F distribution and the exact design. Negative values can occur when the error variance is larger than the subject variance, and they are usually reported as zero.

The Cronbach’s alpha calculator measures the internal consistency of a scale, and the Cohen’s kappa calculator measures agreement for categories.

Frequently asked questions

What does the intraclass correlation measure?

It measures how much of the variation in ratings is due to real differences between subjects rather than to differences between raters or random error. A value near 1 means the raters agree, and a value near 0 means the ratings are mostly noise.

Which of the ICC forms should I use?

Choose by design. Use ICC(1) if each subject is rated by a different random set of raters, ICC(2) if the same raters rate all subjects and are a sample of possible raters, and ICC(3) if these particular raters are the only ones of interest. Use the single-rater form if one rater will be used in practice, and the mean form if you will average several.

What is the difference between absolute agreement and consistency?

Absolute agreement requires the raters to give the same scores. Consistency only requires them to rank subjects the same way, so one rater who always scores two points higher still counts as consistent. Agreement is the stricter standard and is preferred when scores are compared across raters.

What values are acceptable?

A common guide calls values below 0.5 poor, 0.5 to 0.75 moderate, 0.75 to 0.9 good and above 0.9 excellent. The level you need depends on how the scores are used. Clinical decisions about individuals need higher reliability than comparisons of group averages.

Was this calculator helpful?

Thanks for your feedback!

Results are estimates provided for information purposes. They do not replace professional advice.