Intraclass Correlation Coefficient (ICC): Measure Rating Reliability

Two doctors rate the same 50 patients on a severity scale. Do they agree? Three teachers grade the same set of essays. Are the scores consistent? A factory runs two instruments on the same batch. Do the readings match? The Intraclass Correlation Coefficient (ICC) answers these questions with a single number: an ICC close to 1 means almost all the variation comes from real differences between subjects, while an ICC close to 0 means the raters add as much noise as the subjects add signal. Upload a CSV to run an ICC analysis.

Worked Example (Simulated): How Well Do Three Raters Agree?

Question: How well do the ratings agree when the goal is to use one rating from a rater drawn from a wider pool?

Inputs: Simulated data, seed 417. The script makes ratings for 24 subjects from 3 raters, for 72 ratings in all. Each score is 50 plus a subject effect, a rater effect, and random error. The script sets subject spread 8, rater spread 2 and error spread 4.

Method: Use ICC(2,1), a two-way random-effects form for absolute agreement, meaning the same scores, for one rating. The script finds the subject, rater, and error mean squares, then uses them in the ICC formula. Because the spreads are built in, the true ICC(2,1) is 64 / (64 + 4 + 16) = 0.76.

import numpy as np

# Simulated data, seed 417
rng = np.random.default_rng(417)
n_subjects = 24
n_raters = 3
sd_subject, sd_rater, sd_error = 8, 2, 4
subject_effect = rng.normal(0, sd_subject, n_subjects)
rater_effect = rng.normal(0, sd_rater, n_raters)
error = rng.normal(0, sd_error, (n_subjects, n_raters))
scores = 50 + subject_effect[:, None] + rater_effect[None, :] + error

n, k = scores.shape
grand_mean = scores.mean()
subject_means = scores.mean(axis=1)
rater_means = scores.mean(axis=0)
ss_subject = k * np.sum((subject_means - grand_mean) ** 2)
ss_rater = n * np.sum((rater_means - grand_mean) ** 2)
residual = scores - subject_means[:, None] - rater_means[None, :] + grand_mean
ss_error = np.sum(residual ** 2)
ms_subject = ss_subject / (n - 1)
ms_rater = ss_rater / (k - 1)
ms_error = ss_error / ((n - 1) * (k - 1))
icc = (ms_subject - ms_error) / (
    ms_subject + (k - 1) * ms_error + k * (ms_rater - ms_error) / n
)
# True ICC(2,1) from the spreads built into the data.
var_s, var_r, var_e = sd_subject**2, sd_rater**2, sd_error**2
true_icc = var_s / (var_s + var_r + var_e)

print('Simulated data, seed 417')
print(f'Subjects: {n}; raters: {k}; ratings: {scores.size}')
print('Model: ICC(2,1), two-way random effects, absolute agreement, single rating')
print(f'Mean squares: subjects={ms_subject:.3f}, raters={ms_rater:.3f}, '
      f'error={ms_error:.3f}')
print(f'ICC(2,1): {icc:.3f}')
print(f'True ICC(2,1) built into the data: {true_icc:.3f}')
print(f'Score range: {scores.min():.2f} to {scores.max():.2f}')

Printed output:

Simulated data, seed 417
Subjects: 24; raters: 3; ratings: 72
Model: ICC(2,1), two-way random effects, absolute agreement, single rating
Mean squares: subjects=257.796, raters=54.329, error=12.780
ICC(2,1): 0.849
True ICC(2,1) built into the data: 0.762
Score range: 28.43 to 68.63

Result: ICC(2,1) is 0.849 in this simulated set, which is high agreement under the stated model. The true value built into the data is 0.762, so the run's 0.849 is above it. With 24 subjects, one run can land this far off. The score range is 28.43 to 68.63.

Limit: This is a simulated set with a chosen model. It does not show how well real raters or a real scale will agree. The result has no confidence interval, a range that shows uncertainty.

What Is the Intraclass Correlation Coefficient?

The ICC measures how consistent ratings are when multiple raters (or instruments, or time points) evaluate the same set of subjects. Unlike a simple correlation between two raters, ICC handles any number of raters simultaneously and distinguishes between two fundamentally different questions: do the raters rank subjects in the same order (consistency), and do they assign the same actual scores (agreement)?

The intuition is straightforward. ICC partitions the total variance in your ratings into three buckets: variance due to real differences between subjects (the signal you want), variance due to differences between raters (systematic bias), and residual noise (random disagreement). An ICC close to 1.0 means almost all the variance comes from real subject differences, raters agree strongly. An ICC close to 0 means the raters contribute as much noise as the subjects contribute signal. The ratings are unreliable.

Consider a concrete example. Five psychologists independently rate 100 patients on depression severity using a 0-100 scale. If the psychologists tend to give similar scores to the same patient: Patient A gets scores of 72, 75, 70, 73, and 71 from the five raters. That is high agreement and the ICC will be high. But if Patient A gets scores of 40, 85, 55, 90, and 30, the raters disagree wildly and the ICC will be low, even if each rater individually uses the full scale.

ICC is not the same as Pearson correlation. Two raters can correlate at r = 0.95 while one consistently scores 20 points higher than the other. Correlation only captures the ranking relationship; it misses systematic bias entirely. ICC catches both. If you need to know whether raters agree on actual values (not just relative ordering), ICC is the right measure.

Real-World Applications

ICC shows up wherever multiple people or instruments measure the same thing, and you need to know whether those measurements are trustworthy.

Inter-rater reliability in clinical settings. Do two doctors agree on diagnoses? In mental health, multiple clinicians often rate patient severity on standardized scales like the Hamilton Depression Rating Scale or the PHQ-9. Before you can use those ratings to make treatment decisions, you need to know whether different clinicians would reach the same conclusions. A low ICC suggests that a meaningful share of the score variation comes from the rater rather than the patient's condition. Judge any ICC against the stakes of the decision and the confidence interval.

Test-retest reliability in surveys and assessments. Does the same survey give consistent results when administered twice? If you measure employee engagement this month and again next month (assuming nothing changed), the scores should be similar. ICC quantifies that stability.

Measurement reliability across instruments. Do two instruments agree? A manufacturing lab might use two spectrophotometers to measure the same samples. A hospital might compare blood pressure readings from two devices. ICC tells you whether the instruments are interchangeable or whether switching instruments introduces meaningful measurement error.

Performance review consistency. When three managers rate the same set of employees, are the ratings fair and consistent? ICC reveals whether one manager systematically scores higher than the others (bias) and whether the managers agree on who the top and bottom performers are (consistency). Companies use this to audit their review processes and identify managers who need calibration training.

Content moderation and quality scoring. Online platforms have teams of reviewers scoring content for quality, toxicity, or relevance. ICC tells you whether the scoring rubric is being applied consistently. Low ICC means the rubric is ambiguous or the reviewers need training.

What Data Do You Need?

You need a CSV in long format with one row per subject-rater combination. At minimum, you need three columns: a subject identifier (patient ID, employee ID, sample number), a rater identifier (clinician name, manager ID, instrument label), and a numeric rating (severity score, performance score, measurement value). An optional fourth column can define subgroups (diagnostic category, department, product line) for stratified analysis.

For example, if 5 raters each score 100 subjects, your CSV will have 500 rows, one row for each rater-subject pair. The columns might be patient_id, clinician_id, severity_score, and diagnosis_category.

More subjects may improve precision, but there is no fixed sample rule here. Report the confidence interval and judge it in light of the decision. With few subjects, confidence intervals can be wide enough that good and poor reliability are hard to tell apart.

ICC is built for continuous or ordinal rating scales with several levels. If your ratings are binary (yes/no, pass/fail), use Cohen's kappa or Fleiss' kappa instead. If your scale has only 2-3 levels, the ICC estimate may be unstable and kappa-based measures may fit better.

How to Read the Report

ICC Summary Metrics

The report opens with the headline number: the recommended ICC value, its 95% confidence interval, the F-statistic, and the standard error of measurement (SEM). The ICC is interpreted on a scale from 0 to 1, but an estimate can come out negative (usually when rater-to-rater variation exceeds the variation between subjects), and a negative value is read as no reliability. The right form of ICC (consistency or absolute agreement, single or average raters) depends on your design. Any label such as "good" should be judged in context and against the intended use. The confidence interval matters: an ICC of 0.72 with a CI of 0.65-0.78 is a narrow range, but an ICC of 0.72 with a CI of 0.45-0.88 might be anywhere from low to high, and you need more data.

ICC Forms

This page's example uses a two-way random-effects model, absolute agreement, for a single rating. Other designs give other ICC values for the same ratings. The report shows the form it used, with a confidence interval.

Stratified ICC by Category

If you provide a grouping variable (like diagnostic category or department), the report computes ICC within each subgroup. This is often where the most actionable insights appear. Your overall ICC might be 0.82, but the stratified view might reveal that anxiety disorder ratings have ICC of only 0.58 while depression ratings reach 0.86. That tells you exactly where to focus rater training. The bar chart with error bars makes it easy to spot categories where reliability drops below your threshold.

Rater Correlation Heatmap

The heatmap shows pairwise correlations between all raters. Each cell indicates how closely two raters track each other. A uniformly warm heatmap (all correlations high) means the raters rank subjects similarly. It does not show absolute agreement: raters can correlate strongly while one scores systematically higher, which the ICC captures and a correlation does not. A cold row or column points to a rater who disagrees with the group: either due to training gaps, different interpretation of criteria, or systematic bias. Suspiciously perfect agreement (correlations near 1) between specific pairs may indicate that raters collaborated or shared notes, which violates the independence assumption.

Bland-Altman Agreement Plot

The Bland-Altman plot shows the difference between rater pairs plotted against their average. If raters agree, points cluster around zero with no pattern. A cluster shifted above or below zero reveals systematic bias, one rater scores consistently higher. A funnel shape (wider spread at higher values) means agreement is worse for more severe cases. This plot shows agreement directly: you can read off the limits of agreement (mean difference plus or minus 1.96 standard deviations) and decide whether the disagreement is clinically or practically meaningful.

Rater Summary Table

The rater summary table shows each rater's mean score, standard deviation, and systematic bias relative to the group average. If Rater 3 averages 75 while the group averages 65, that is a +10 systematic bias. The table makes it easy to identify lenient and strict raters. This is especially valuable for HR teams auditing performance reviews or clinical directors overseeing assessment programs.

Model Comparison Table and Category Reliability Table

The model comparison table presents ICC values for the recommended models side by side: agreement ICC (which penalizes systematic bias between raters) versus consistency ICC (which ignores it). If the two differ substantially, systematic bias is present and needs to be addressed through calibration. The category reliability table shows ICC, sample size, and confidence intervals for each subgroup, making it easy to flag categories that fall below your reliability threshold.

Interpretation and Recommendations

The final section translates all the numbers into plain language: which ICC model was selected and why, whether reliability meets your threshold, which raters or categories need attention, and specific recommendations for improvement. This is the card to share with stakeholders who do not need the statistical details.

Choosing the Right ICC Model

The most common mistake in ICC analysis is choosing the wrong model. The choice comes down to two questions.

Are your raters a random sample from a larger population? If you are studying inter-rater reliability for a clinical scale and your five clinicians represent "any trained clinician who might use this scale," a model that treats raters as a random sample lets you generalize beyond the specific raters in your study. This page's example uses that kind of model. If only these five clinicians will ever use the scale, a model that treats the raters as fixed fits better.

Do you care about absolute agreement or just consistency? Agreement ICC penalizes systematic differences between raters. If Rater A averages 80 and Rater B averages 60, the agreement ICC will be lower than the consistency ICC. Use agreement when the actual score matters (clinical diagnosis, certification testing). Use consistency when only the ranking matters (relative performance comparisons).

Single measure or average? Single-measure ICC tells you how reliable one rater's score is. Average-measure ICC tells you how reliable the mean of all raters' scores is. If your decision will be based on a single rater's score (e.g., a patient sees one clinician), report single-measure ICC. If it will be based on the average of multiple raters (e.g., a panel of judges), report average-measure ICC.

When to Use Something Else

ICC is powerful but not universal. Several situations call for a different tool.

If your ratings are categorical rather than numeric (pass/fail, present/absent, category A/B/C) use Cohen's kappa (for two raters) or Fleiss' kappa (for three or more). Kappa is designed for nominal and ordinal categories. Applying ICC to binary data produces unstable and misleading results.

If you want to compare two measurement methods (not raters, but instruments or techniques) and visualize where they agree and disagree, Bland-Altman plots are the standard approach. The ICC report includes Bland-Altman plots as one of its layout cards, but if method comparison is your primary question, a dedicated Bland-Altman analysis gives you limits of agreement and proportional bias testing in more detail.

If you are assessing the internal consistency of a multi-item questionnaire use Cronbach's alpha. Alpha measures consistency among items, not among raters. It answers a fundamentally different question: is the questionnaire itself reliable, rather than are the people using it consistent.

If you have only two raters and a continuous scale, a simple Pearson correlation paired with a Bland-Altman plot may be sufficient. ICC still works with two raters, but the simpler tools are easier to interpret and communicate when the study design is straightforward.

Common Pitfalls

Using correlation instead of ICC. Pearson correlation measures linear association but ignores systematic bias. Two raters can correlate r = 0.90 while one consistently scores 20 points higher than the other. Correlation looks high, while ICC for absolute agreement is lower. Use ICC when absolute agreement matters.

Ignoring the confidence interval. An ICC of 0.70 sounds good, but if the 95% CI is 0.40-0.90, you cannot rule out poor reliability. With few subjects, confidence intervals can be very wide. Report the CI alongside the point estimate, and judge its width against the decision.

Treating ordinal ratings as continuous without checking. ICC assumes equal intervals between rating levels. A 5-point Likert scale treated as continuous is often treated as continuous, but a 3-point scale may be too coarse for stable ICC estimates. With few levels, consider weighted kappa.

The R Code Behind the Analysis

Every report includes the exact R code used to produce the results: reproducible, auditable, and citable. The analysis uses the irr package's icc() function to compute ICC values with confidence intervals. Rater correlation heatmaps use base R's cor(), Bland-Altman plots use custom calculations of mean difference and limits of agreement, and stratified ICC is computed by subsetting the data by category and running icc() on each subset. No custom implementations, no black boxes. Every step is visible in the code tab of your report.