The intraclass correlation coefficient, or ICC, measures the reliability of ratings or measurements made by several raters. The intraclass correlation calculator takes the mean squares from a two-way ANOVA, the number of subjects and the number of raters, and gives ICC(1,1), ICC(2,1), ICC(3,1), ICC(2,k) and ICC(3,k).
How the calculation works
- Run a two-way analysis of variance on the ratings, with subjects as rows and raters as columns. Read the mean squares for subjects (MSR), raters (MSC) and error (MSE).
- ICC(3,1) = (MSR − MSE) ÷ (MSR + (k − 1) × MSE) measures consistency for a single rater, where k is the number of raters.
- ICC(2,1) = (MSR − MSE) ÷ (MSR + (k − 1) × MSE + k × (MSC − MSE) ÷ n) measures absolute agreement, and it is lower when raters differ systematically.
- ICC(1,1) uses the within-subject mean square MSW = (MSC + (n − 1) × MSE) ÷ n. The versions for the mean of k raters are ICC(2,k) = (MSR − MSE) ÷ (MSR + (MSC − MSE) ÷ n) and ICC(3,k) = (MSR − MSE) ÷ MSR.
Example
Three raters score 20 subjects. The ANOVA gives MSR = 52.6, MSC = 4.1 and MSE = 3.2. Then ICC(3,1) = 49.4 ÷ 59.0 = 0.837, and ICC(2,1) = 49.4 ÷ 59.135 = 0.835. ICC(1,1) is 0.835, and the reliability of the mean of the three raters is 0.938 for absolute agreement and 0.939 for consistency. A single rater is in the good range.
Why it matters
Before using a scale, a test or a panel of raters, you need to know whether the scores can be trusted. The ICC answers that with a single number, and the five forms match the common study designs, so it can be reported in the way journals expect.
Good to know
The formulas follow Shrout and Fleiss and use mean squares from an ANOVA without missing data. They give point estimates and not confidence intervals, which need the F distribution and the exact design. Negative values can occur when the error variance is larger than the subject variance, and they are usually reported as zero.
Related tools
The Cronbach’s alpha calculator measures the internal consistency of a scale, and the Cohen’s kappa calculator measures agreement for categories.