What ICC is for

The intraclass correlation coefficient answers one family of questions: can I trust a measurement that passes through people or instruments? Whenever the same thing is measured more than once (by two raters, two devices, or the same instrument on two occasions), the measurements will not agree perfectly. ICC quantifies how much of the variation in your data reflects real differences between the things being measured, versus noise introduced by the measuring itself.

It is not a prediction. It is a verdict on the measuring, and every analysis you run downstream inherits that verdict. A driver analysis, a group comparison, or a forecast built on unreliable scores is quietly built on sand; ICC is how you check the foundation first.

Common applications

Field The question ICC answers
Clinical research Do two radiologists read the same scan the same way? (inter-rater reliability)
Psychometrics & surveys Does the instrument give the same score on a retest? (test–retest reliability)
Machine learning Can I rely on the labels my annotation team produced? (label-quality audit)
Quality control Do two gauges, labs, or operators agree on the same parts? (method agreement)
People operations Do interviewers or QA reviewers score candidates/calls consistently?

What you need as input

One table, wide format: a row per subject (the things being measured), a column per rater or measurement occasion, numeric scores in the cells.

  • Scores should be continuous or at least ordered with enough distinct values to treat as numeric. (Binary yes/no ratings are better served by kappa and McNemar; ICC variants for dichotomous data exist, but kappa is the standard there.)
  • Design matters more than size: know whether the same raters scored every subject (a crossed design) or each subject got different raters. That is Question 1 of the chooser below.
  • Size still matters: confidence intervals with a dozen subjects are wide, as this example will demonstrate honestly. Koo & Li (2016) suggest at least 30 subjects and 3 raters for stable estimates; below that, report the interval and let it do the talking.

What you get as output

One number per ICC form, usually between 0 and 1 (sample estimates can dip below zero when reliability is truly poor), plus a 95% confidence interval and an F-test. The conventional interpretation bands (Koo & Li 2016), applied to the interval, not just the point estimate:

ICC Reliability
< 0.50 Poor
0.50 – 0.75 Moderate
0.75 – 0.90 Good
> 0.90 Excellent

The six forms, and the three questions that choose between them

The Shrout & Fleiss (1979) lineup has six forms: three models × single-or-average measures. Three questions select yours:

  1. Did the same raters score every subject? No → one-way model, ICC(1). Rater habits cannot be separated from noise. Yes → two-way model; continue.
  2. Do rater differences in level count as error? If a 70 from one rater must mean what a 70 means from another (scores travel between raters), you need absolute agreement: ICC(2,1). If only ranking matters and level washes out, you need consistency: ICC(3,1). (Statisticians file the 2-vs-3 choice under whether raters are a random sample or a fixed panel; McGraw & Wong 1996 decouple the axes. For picking the number, agreement versus consistency is what decides it.)
  3. Will decisions use a single rater’s score, or the panel average? Single → ICC(·,1). Average of k raters → ICC(·,k), which is always at least as high whenever there is real reliability to amplify.

The worked example

The data and its provenance

Twelve subjects, each scored by the same three raters. The dataset is synthetic and fully reproducible: generated by icc_example.py (seed 42) with true subject scores ~ N(70, 10), additive rater biases of 0, +10, and −3 points, and rating noise ~ N(0, 3). Because we built it, we know the ground truth the estimators are chasing, which is the point of a worked example.

ratings <- read.csv("ratings.csv")
ratings
##    rater1 rater2 rater3
## 1    73.2   86.4   71.4
## 2    57.0   70.7   53.7
## 3    80.1   87.4   73.9
## 4    77.4   93.1   75.9
## 5    49.2   59.4   49.1
## 6    58.1   68.2   55.3
## 7    77.7   80.1   66.7
## 8    64.4   78.7   67.2
## 9    69.5   77.3   64.4
## 10   63.4   73.7   60.1
## 11   76.8   89.5   76.1
## 12   78.4   90.4   75.4

First look: the raters do not agree on level

colMeans(ratings)
##   rater1   rater2   rater3 
## 68.76667 79.57500 65.76667
matplot(ratings, type = "b", pch = 16, lty = 1,
        xlab = "Subject", ylab = "Score",
        main = "Three raters, twelve subjects")
legend("topright", legend = names(ratings), col = 1:3, pch = 16, bty = "n")

Rater 2 runs about 10.8 points hotter than rater 1 on the same subjects: visibly parallel profiles, shifted in level. Whether that shift counts as error is exactly what separates the ICC forms.

Every form at once

res <- ICC(ratings)
res$results
##                          type       ICC         F df1 df2            p
## Single_raters_absolute   ICC1 0.5734119  5.032545  11  24 4.616760e-04
## Single_random_raters     ICC2 0.6225058 58.926373  11  22 1.032616e-13
## Single_fixed_raters      ICC3 0.9507602 58.926373  11  22 1.032616e-13
## Average_raters_absolute ICC1k 0.8012934  5.032545  11  24 4.616760e-04
## Average_random_raters   ICC2k 0.8318520 58.926373  11  22 1.032616e-13
## Average_fixed_raters    ICC3k 0.9830297 58.926373  11  22 1.032616e-13
##                         lower bound upper bound
## Single_raters_absolute    0.2396792   0.8330165
## Single_random_raters      0.0429272   0.8918586
## Single_fixed_raters       0.8763540   0.9842429
## Average_raters_absolute   0.4860469   0.9373663
## Average_random_raters     0.1185993   0.9611522
## Average_fixed_raters      0.9550820   0.9946919

Reading the table against the design

Our design: the same panel rated everyone (two-way), scores need to travel between raters (absolute agreement), and decisions ride on single ratings (single measure). The three questions select ICC(2,1):

icc21 <- res$results[res$results$type == "ICC2", ]
icc21
##                      type       ICC        F df1 df2            p lower bound
## Single_random_raters ICC2 0.6225058 58.92637  11  22 1.032616e-13   0.0429272
##                      upper bound
## Single_random_raters   0.8918586

Point estimate 0.62, 95% CI [0.04, 0.89]. With n = 12 subjects the interval is wide, and per Koo & Li the interval, not the point, should drive the label.

The consistency form on the identical data:

icc31 <- res$results[res$results$type == "ICC3", ]
round(c(ICC = icc31$ICC, lower = icc31$"lower bound", upper = icc31$"upper bound"), 3)
##   ICC lower upper 
## 0.951 0.876 0.984

Same 36 numbers: agreement 0.62, consistency 0.95. Neither is wrong; they answer different questions.

The calibration experiment: the gap is a diagnosis, not a verdict

If the agreement/consistency gap is driven by rater level bias, then removing each rater’s mean offset should send agreement up toward the consistency value. Center each rater and recompute:

centered <- sweep(ratings, 2, colMeans(ratings)) + mean(as.matrix(ratings))
res_c <- ICC(centered)
res_c$results[res_c$results$type %in% c("ICC2", "ICC3"),
              c("type", "ICC", "lower bound", "upper bound")]
##                      type       ICC lower bound upper bound
## Single_random_raters ICC2 0.9547399   0.8882823   0.9854327
## Single_fixed_raters  ICC3 0.9547399   0.8858741   0.9855434

After calibration, agreement ICC(2,1) rises to 0.95, effectively the consistency ceiling. The gap between the two forms measured the level bias and told us the fix (calibrate raters), not the verdict (distrust the instrument).

Averaging: the panel is more reliable than any member

res$results[res$results$type %in% c("ICC2k", "ICC3k"),
            c("type", "ICC", "lower bound", "upper bound")]
##                        type       ICC lower bound upper bound
## Average_random_raters ICC2k 0.8318520   0.1185993   0.9611522
## Average_fixed_raters  ICC3k 0.9830297   0.9550820   0.9946919

If decisions will only ever use the three-rater average, ICC(2,k) applies. But report the form you will actually live with downstream.

Cross-check against an independent implementation

The video’s on-screen numbers were computed independently in Python (ANOVA mean squares, no libraries). Agreement between implementations:

manual <- c(ICC1 = 0.57, ICC2 = 0.62, ICC3 = 0.95, ICC2k = 0.83, ICC3k = 0.98)
r_vals <- round(res$results[match(c("ICC1","ICC2","ICC3","ICC2k","ICC3k"),
                                  res$results$type), "ICC"], 2)
data.frame(form = names(manual), python = as.numeric(manual), R = r_vals,
           match = as.numeric(manual) == r_vals)
##    form python    R match
## 1  ICC1   0.57 0.57  TRUE
## 2  ICC2   0.62 0.62  TRUE
## 3  ICC3   0.95 0.95  TRUE
## 4 ICC2k   0.83 0.83  TRUE
## 5 ICC3k   0.98 0.98  TRUE

Honest limitations

  • Small samples: our n = 12 produces the CI [0.04, 0.89] you saw above, nearly the whole ruler. Report intervals, and prefer ≥30 subjects when you design the study.
  • Dependence and structure: strongly dependent ratings, nested designs, or missing cells push you toward mixed-effects formulations rather than the closed forms shown here.
  • Negative estimates happen in real samples when reliability is truly poor; they are a finding, not a bug.
  • ICC is scale-dependent: restrict the range of subjects and ICC drops even with identical raters, so compare ICCs only across comparable populations.

References

  • Shrout, P.E. & Fleiss, J.L. (1979). Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428.
  • McGraw, K.O. & Wong, S.P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.
  • Koo, T.K. & Li, M.Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163.

Session

sessionInfo()$R.version$version.string
## [1] "R version 4.5.1 (2025-06-13)"
packageVersion("psych")
## [1] '2.6.1'
Want to run this analysis on your own data? Upload CSV — Free Analysis See Pricing