Which ICC do you actually need?
The reproducible worked example behind our ICC lesson: the same ratings scored by different ICC forms, with confidence intervals.
The data
Twelve subjects rated by three raters. The same twelve rows, scored by every ICC form, cross-checked against numpy, R's psych::ICC, and an independent recompute, agreeing to three decimal places.
The headline numbers
- ICC(1,1)0.573
- ICC(2,1)0.623
- ICC(3,1)0.951
- ICC(2,k)0.832
- ICC(3,k)0.983
One dataset returns 0.573 or 0.983 depending only on which ICC form you ask for. That is not a rounding difference, it is the gap between poor and excellent. The form is a modelling choice about what you are generalising to, not a formula preference.
Download and re-run it
Every figure in the document is derived when it is knitted, not typed in. Re-run the source and you should get the same numbers, or we have a problem worth hearing about.
- icc.html · The knitted document1.4 MB
- icc.Rmd · R Markdown source9 KB
- ratings.csv · The ratings1 KB
- icc_example.py · Generates the data2 KB
The honest limit
A point estimate without its confidence interval is not a reliability claim, and with a dozen subjects those intervals are wide. The document says so rather than quietly reporting three decimal places. The product's own report was missing intervals when this was written, which we filed against ourselves and fixed.