Rater Reliability (ICC)
Shows how reliable numeric ratings are across raters or repeated measurements, every common intraclass correlation with its likely range and when to use it, and which raters score systematically higher or lower.
RUN DATE · 14 September 2026
DATA · 120 rows
How reliable are our four inspectors' quality scores, and which reliability figure should we report?
- SummaryThe headline reliability for one rater and for the average of raters.
- Every ICC with its rangeEach reliability coefficient with its likely range.
- Where the variation comes fromHow much of the variation is real differences, rater bias and noise.
- Lenient and severe ratersHow far each rater scores above or below the consensus on the same subjects.
- Every rating against the consensusEach rating beside its subject's average, coloured by rater.
- Which ICC to reportEvery form with its range, band and when to use it.
- Where raters disagree mostThe subjects with the widest spread of ratings.
- What the results rely onEach condition the reliability figures depend on, and whether it holds.
- How it was doneThe method, the data used, and what to keep in mind.
How reliable the ratings are
Single raters good, average forms stronger
Single rater forms reach good reliability; average forms sit clearly higher and reach excellent reliability.
The single rater forms cluster in the good band; the average forms sit well above them in the excellent band.
Single raters good, average forms stronger
The single rater forms (agreement, consistency, one-way) all have lower bounds in the good band and intervals that stay within or just touch the excellent band. The average-of-raters forms sit clearly higher, with lower bounds solidly in the excellent band. One rater differs systematically in level, which means agreement forms sit slightly below their consistency counterparts, but this does not prevent the single rater forms from reaching good reliability. Which form to report depends on whether you will use single rater or average scores in practice.
The single rater forms cluster in the good band; the average forms sit well above them in the excellent band.
Subjects dominate, raters visible but small
Real differences between inspectors' subjects dominate clearly, though rater bias is noticeable alongside noise.
The subjects bar towers over raters and noise bars, showing strong separation of inspection targets.
Subjects dominate, raters visible but small
Subjects account for the vast majority of variation, clearly dominating the raters and noise components. The raters bar is noticeable, signaling systematic differences between inspectors that warrant reporting the agreement form alongside overall reliability. Noise remains modest. However, the strained checks on subject count and completeness of ratings limit confidence in these shares, and the violated check on rater neutrality confirms that inspector bias exists and shapes the raters component.
The subjects bar towers over raters and noise bars, showing strong separation of inspection targets.
Where raters differ
Dr. Chen severe, Dr. Baker lenient, Dr. Adams mixed
Dr. Chen scores systematically lower, Dr. Baker higher; both intervals clear zero. Dr. Adams and Dr. Diaz straddle zero, showing no systematic offset.
Dr. Chen's interval sits entirely below zero; Dr. Baker's sits entirely above zero. Dr. Adams and Dr. Diaz intervals cross zero.
Dr. Chen severe, Dr. Baker lenient, Dr. Adams mixed
Dr. Chen's offset clearly falls below consensus, while Dr. Baker's clearly rises above it. Their intervals do not overlap, marking them as the most severe and most lenient raters. Dr. Adams and Dr. Diaz show mixed patterns, with intervals that cross zero and offer no evidence of systematic bias. This systematic variation among raters violates the assumption that no rater is systematically higher or lower, a check that matters for reliability claims.
Dr. Chen's interval sits entirely below zero; Dr. Baker's sits entirely above zero. Dr. Adams and Dr. Diaz intervals cross zero.
Dr. Baker systematically scores higher than peers
On this card's data, Dr. Chen scores below the consensus, not close to it.
The diagonal line from lower left to upper right; Dr. Baker's colour sits visibly above it across the range.
The numbers
The agreement form for one rater is in the good band
Single rater scores show good reliability, but average of all raters rises to excellent, clearly stronger.
Single rater forms stay in good band while all average forms reach excellent band, a consistent and sharp split across the table.
The agreement form for one rater is in the good band
ICC(2,1) agreement for a single rater falls in the good band, while ICC(2,k) for the average of all raters jumps to the excellent band. This shift is clear and substantial across the table. However, a rater bias check is violated, meaning inspectors do not score uniformly relative to each other; this limits how much confidence to place in any single rater's scores alone. If your study design uses all raters' average, report ICC(2,k) agreement; if single raters must substitute for each other, report ICC(2,1) and acknowledge the good but not excellent reliability.
Single rater forms stay in good band while all average forms reach excellent band, a consistent and sharp split across the table.
The numbers
Two subjects show much wider disagreement
S15 and S09 show wider disagreement than others, signalling reliability problems for those specific subjects.
S15 and S09 diverge sharply from the tight cluster of S11 through S14, breaking the otherwise gradual pattern.
Two subjects show much wider disagreement
S15 and S09 stand out clearly with spreads well above the rest, while S11 through S14 cluster tightly at the lower end. This pattern is sharp enough to act on: those two subjects need investigation. However, one rater systematically rates higher or lower than peers, which complicates whether the spreads reflect true subject ambiguity or rater bias.
S15 and S09 diverge sharply from the tight cluster of S11 through S14, breaking the otherwise gradual pattern.
Assumptions
Checks: one violated, two strained, two hold
Violated: no rater systematically higher or lower; strained: enough subjects, every subject rated by every rater.
Holding: enough raters, residuals roughly normal.
Checks: one violated, two strained, two hold
Enough subjects is strained: fewer than about 30 subjects leaves every ICC with a wide interval. Every subject rated by every rater is strained: the two-way forms use only subjects every rater scored; excluded subjects can bias the result. No rater systematically higher or lower is violated: a real rater effect lowers agreement ICC(2) below consistency ICC(3): report the form that matches how the scores will be used. The other two hold.
Holding: enough raters, residuals roughly normal.
How it was done
Intraclass correlation from two-way analysis of variance mean squares (Shrout and Fleiss 1979; intervals for the agreement forms by McGraw and Wong 1996) on Quality Score: 28 Sample ID values each rated by all 4 Rater Name values (112 ratings used of 120 rows); excluded: 1 row with a blank Sample ID, Rater Name or Quality Score; 1 repeated rating of the same Sample ID by the same Rater Name averaged into one; 2 Sample ID values not rated by every Rater Name (S12, S30); not used: Session (reliability needs only the subject, rater and score); 95% intervals from the F distribution; bands by Koo and Li (2016): below 0.5 poor, 0.5 to 0.75 moderate, 0.75 to 0.9 good, above 0.9 excellent; the rater effect tested by the rater mean square against the residual mean square.
112 of 120 rows · Sample ID, Rater Name, Quality Score → 4 raters of 28 subjects
caveatRaters differed systematically in level; two samples lacked complete ratings across all four inspectors.
The analysis used a two-way analysis of variance method to compute intraclass correlations on quality scores from four raters assessing samples. Eight ratings were excluded or not used: one row with missing data, one duplicate rating that was averaged, and two samples that lacked ratings from every inspector. The rater effect test found that inspectors differed systematically in their scoring level, which violates a key assumption. Additionally, the sample size was strained relative to the method's needs, and two samples remained incomplete despite the design calling for every rater to score every sample. This systematic difference in rater level could bias the reliability estimate and should be checked before choosing which ICC form to report.
The code behind this report
The code that produced every figure in this report, exactly as it ran. Fingerprint 1412449386d36828. The same code on the same data gives the same report.
`standard_icc_v2` <- function(pf) {
`%||%` <- function(a, b) if (!is.null(a)) a else b
#' Readable figures (LAT-3180, LAT-3181): whole numbers from a thousand up, one decimal from a hundred, two from
#' one, three significant figures below one. A cell carries what the value needs, not what R prints.
tidy <- function(x) {
x <- as.numeric(x)
ifelse(is.na(x), NA_real_,
ifelse(abs(x) >= 1000, round(x, 0),
ifelse(abs(x) >= 100, round(x, 1),
ifelse(abs(x) >= 1, round(x, 2), signif(x, 3)))))
}
#' P-values below 0.0001 leave as a floor (LAT-3181): the four-decimal serializer rounds 1.4e-05 to 0.
p_cell <- function(p) { p <- as.numeric(p); ifelse(is.na(p), NA_real_, ifelse(p < 1e-4, 1e-12, signif(p, 3))) }
inputs <- pf$taskList$inputs
params <- inputs$module_parameters %||% list()
# THE QUESTION this tool answers: the customer's objective, verbatim, when given.
question <- (inputs$userContext %||% list())$objective %||%
"How reliable are these ratings: do the raters agree, and which reliability coefficient should I report?"
#' ## Column mapping
#' Long format, one row per rating: `subject` (the thing rated), `rater` (who or what rated it) and `score` (the
#' numeric rating). Semantic names inside; the customer's own headers live in `col_map`.
col_map <- inputs$column_mapping %||% list()
df <- renderObject.taskFunction.init(inputs, col_map) # df has SEMANTIC names
human <- function(sem) {
v <- col_map[[sem]]
if (is.null(v) || !nzchar(as.character(v))) sem else as.character(v)
}
subject_name <- human("subject"); rater_name <- human("rater"); score_name <- human("score")
for (sem in c("subject", "rater", "score"))
if (!(sem %in% names(df))) stop(sprintf("column_mapping must map '%s' (%s was not found).", sem, human(sem)))
n_in <- nrow(df)
#' ## The columns this tool did not look at, read from the raw rows (init narrows `df` to the mapped columns)
raw_names <- local({
ds <- inputs$dataset %||% inputs$df
if (is.data.frame(ds)) return(names(ds))
if (is.list(ds) && length(ds) > 0) {
rows <- ds[seq_len(min(length(ds), 50))]
nm <- unique(unlist(lapply(rows, function(r) if (is.list(r)) names(r) else NULL)))
if (length(nm)) return(nm)
if (!is.null(names(ds)) && all(nzchar(names(ds)))) return(names(ds))
}
character(0)
})
mapped_actual <- unique(as.character(unlist(col_map)))
ignored_cols <- setdiff(raw_names, unique(c(mapped_actual, make.names(mapped_actual))))
#' ## Data preparation
#' The score is read as a number (95% rule). Rows with a blank subject, rater or score are excluded and counted;
#' repeated ratings of one subject by one rater are averaged into one and counted. The two-way ICCs need every
#' subject rated by every rater, so a subject missing any rater is excluded and named. At least 5 complete subjects
#' and 2 raters are required; more than 20 raters is refused (the question is then usually a different design).
sc <- df$score
if (!is.numeric(sc)) {
ch <- trimws(as.character(sc)); nb <- !is.na(ch) & ch != ""
conv <- suppressWarnings(as.numeric(ch))
if (sum(nb) == 0 || sum(!is.na(conv[nb])) < 0.95 * sum(nb))
stop(sprintf("The score column '%s' is not numeric: fewer than 95%% of its values read as numbers. An ICC needs numeric ratings; for categories (yes/no, grades as words) use an agreement statistic such as kappa.", score_name))
sc <- conv
}
subj <- trimws(as.character(df$subject)); rat <- trimws(as.character(df$rater))
bad <- is.na(subj) | subj == "" | is.na(rat) | rat == "" | !is.finite(sc)
n_bad <- sum(bad)
subj <- subj[!bad]; rat <- rat[!bad]; sc <- as.numeric(sc[!bad])
cell <- paste(subj, rat, sep = "\r")
n_dup <- sum(duplicated(cell))
agg <- aggregate(sc, by = list(subject = subj, rater = rat), FUN = mean)
names(agg)[3] <- "score"
raters <- sort(unique(agg$rater)); k <- length(raters)
if (k < 2) stop(sprintf("Only %d rater found in '%s'; reliability between raters needs at least 2.", k, rater_name))
if (k > 20) stop(sprintf("%d raters found in '%s'; this tool compares up to 20 raters of the same subjects. Check that '%s' names who rated, not what was rated.", k, rater_name, rater_name))
per_subject <- table(agg$subject)
complete <- names(per_subject)[per_subject == k]
incomplete <- setdiff(names(per_subject), complete)
n <- length(complete)
if (n < 5) stop(sprintf("Only %d %s value(s) were rated by all %d raters; the two-way ICCs need at least 5 complete subjects.", n, subject_name, k))
agg <- agg[agg$subject %in% complete, , drop = FALSE]
Y <- matrix(NA_real_, n, k, dimnames = list(sort(complete), raters))
Y[cbind(match(agg$subject, rownames(Y)), match(agg$rater, colnames(Y)))] <- agg$score
if (isTRUE(stats::var(as.vector(Y)) == 0)) stop(sprintf("Every score in '%s' is the same; there is no variation to attribute to subjects or raters.", score_name))
n_used <- n * k
#' ## Two-way ANOVA mean squares (Shrout and Fleiss 1979; McGraw and Wong 1996)
gm <- mean(Y); rm_ <- rowMeans(Y); cm_ <- colMeans(Y)
SSR <- k * sum((rm_ - gm)^2); SSC <- n * sum((cm_ - gm)^2); SST <- sum((Y - gm)^2)
SSE <- SST - SSR - SSC; SSW <- SST - SSR
MSR <- SSR / (n - 1); MSC <- SSC / (k - 1); MSE <- SSE / ((n - 1) * (k - 1)); MSW <- SSW / (n * (k - 1))
df1 <- n - 1; dfe <- (n - 1) * (k - 1); dfw <- n * (k - 1)
q <- function(a, b) stats::qf(0.975, a, b)
#' ICC(1,1) and ICC(1,k): one-way random (each subject may be rated by different raters)
icc11 <- (MSR - MSW) / (MSR + (k - 1) * MSW); icc1k <- (MSR - MSW) / MSR
F1 <- MSR / MSW; FL1 <- F1 / q(df1, dfw); FU1 <- F1 * q(dfw, df1)
ci11 <- c((FL1 - 1) / (FL1 + k - 1), (FU1 - 1) / (FU1 + k - 1)); ci1k <- c(1 - 1 / FL1, 1 - 1 / FU1)
p1 <- stats::pf(F1, df1, dfw, lower.tail = FALSE)
#' ICC(3,1) and ICC(3,k): two-way mixed, consistency (a rater who is systematically high or low is forgiven)
icc31 <- (MSR - MSE) / (MSR + (k - 1) * MSE); icc3k <- (MSR - MSE) / MSR
F3 <- MSR / MSE; FL3 <- F3 / q(df1, dfe); FU3 <- F3 * q(dfe, df1)
ci31 <- c((FL3 - 1) / (FL3 + k - 1), (FU3 - 1) / (FU3 + k - 1)); ci3k <- c(1 - 1 / FL3, 1 - 1 / FU3)
p3 <- stats::pf(F3, df1, dfe, lower.tail = FALSE)
#' ICC(2,1) and ICC(2,k): two-way random, absolute agreement (raters must match in value), McGraw and Wong interval
icc21 <- (MSR - MSE) / (MSR + (k - 1) * MSE + k * (MSC - MSE) / n)
icc2k <- (MSR - MSE) / (MSR + (MSC - MSE) / n)
a <- k * icc21 / (n * (1 - icc21)); b <- 1 + k * icc21 * (n - 1) / (n * (1 - icc21))
v <- (a * MSC + b * MSE)^2 / ((a * MSC)^2 / (k - 1) + (b * MSE)^2 / dfe)
Fs <- q(df1, v); Fi <- q(v, df1)
lo21 <- n * (MSR - Fs * MSE) / (Fs * (k * MSC + (k * n - k - n) * MSE) + n * MSR)
hi21 <- n * (Fi * MSR - MSE) / (k * MSC + (k * n - k - n) * MSE + n * Fi * MSR)
ci21 <- c(lo21, hi21); ci2k <- c(lo21 * k / (1 + lo21 * (k - 1)), hi21 * k / (1 + hi21 * (k - 1)))
#' The rater effect: does any rater score systematically higher or lower than the others (MSC / MSE)?
Fr <- MSC / MSE; p_rater <- stats::pf(Fr, k - 1, dfe, lower.tail = FALSE)
band <- function(x) ifelse(is.na(x), "not measurable", ifelse(x < 0.5, "poor", ifelse(x < 0.75, "moderate", ifelse(x < 0.9, "good", "excellent"))))
forms <- data.frame(
form = c("ICC(2,1) agreement, one rater", "ICC(3,1) consistency, one rater", "ICC(1,1) one-way, one rater",
"ICC(2,k) agreement, average of raters", "ICC(3,k) consistency, average of raters", "ICC(1,k) one-way, average of raters"),
estimate = c(icc21, icc31, icc11, icc2k, icc3k, icc1k),
low = c(ci21[1], ci31[1], ci11[1], ci2k[1], ci3k[1], ci1k[1]),
high = c(ci21[2], ci31[2], ci11[2], ci2k[2], ci3k[2], ci1k[2]), stringsAsFactors = FALSE)
forms$band <- band(forms$estimate)
forms$use_when <- c("raters are a sample from a larger pool and must give the same values",
"these raters are the only ones of interest and only their ordering must agree",
"each subject may have been rated by different raters",
"the average of all raters' scores will be used, and values must match",
"the average of these raters' scores will be used, and only ordering must match",
"the average score is used and raters differ from subject to subject")
interval_df <- data.frame(form = forms$form, estimate = round(forms$estimate, 3),
low = round(pmax(-1, forms$low), 3), high = round(pmin(1, forms$high), 3), stringsAsFactors = FALSE)
table_df <- data.frame(form = forms$form, estimate = round(forms$estimate, 3), low = round(pmax(-1, forms$low), 3),
high = round(pmin(1, forms$high), 3), band = forms$band, use_when = forms$use_when, stringsAsFactors = FALSE)
#' Each rater's systematic offset: how far their score sits from the subject's consensus (the mean across raters),
#' averaged over the same subjects, with a paired 95% interval. A plain mean score per rater carries the spread
#' between SUBJECTS in its interval, so every rater's interval overlapped even while the rater effect test said the
#' raters differ (LAT-3186, first run). The offset removes the subject and leaves only the rater.
rater_df <- do.call(rbind, lapply(raters, function(r) {
dlt <- Y[, r] - rm_; m <- mean(dlt); h <- stats::qt(0.975, n - 1) * stats::sd(dlt) / sqrt(n)
data.frame(rater = r, offset = tidy(m), ci_low = tidy(m - h), ci_high = tidy(m + h), stringsAsFactors = FALSE)
}))
#' Every rating against its subject's mean across raters: points on the diagonal are raters who agree
set.seed(42)
long <- data.frame(subject_mean = tidy(rep(rm_, times = k)), score = tidy(as.vector(Y)), rater = rep(raters, each = n), stringsAsFactors = FALSE)
if (nrow(long) > 1000) long <- long[sort(sample(nrow(long), 1000)), , drop = FALSE]
rownames(long) <- NULL
#' Where the variation comes from: variance components of the two-way model, as a share of the total
vs <- max(0, (MSR - MSE) / k); vr <- max(0, (MSC - MSE) / n); ve <- max(0, MSE)
tot <- vs + vr + ve
var_df <- data.frame(source = c(sprintf("Subjects (%s)", subject_name), sprintf("Raters (%s)", rater_name), "Unexplained noise"),
share_pct = round(100 * c(vs, vr, ve) / max(tot, 1e-12), 1), stringsAsFactors = FALSE)
#' The subjects the raters disagree about most: the widest spread of scores across raters
rng <- apply(Y, 1, function(x) max(x) - min(x))
top <- order(-rng)[seq_len(min(n, 12))]
disagree_df <- data.frame(subject = rownames(Y)[top], mean = tidy(rm_[top]), lowest = tidy(apply(Y[top, , drop = FALSE], 1, min)),
highest = tidy(apply(Y[top, , drop = FALSE], 1, max)), spread = tidy(rng[top]), stringsAsFactors = FALSE)
rownames(disagree_df) <- NULL
#' ## Assumption checks (LAT-3138)
verdict_p <- function(pv, soft = 0.05, hard = 0.001) if (is.na(pv)) "unknown" else if (pv >= soft) "holds" else if (pv >= hard) "strained" else "violated"
fmt_p <- function(pv) if (is.na(pv)) "" else if (pv < 1e-4) "<0.0001" else as.character(signif(pv, 3))
resid <- as.vector(Y - outer(rm_, rep(1, k)) - outer(rep(1, n), cm_) + gm)
shp <- if (length(resid) >= 3) tryCatch(stats::shapiro.test(if (length(resid) > 5000) resid[seq_len(5000)] else resid)$p.value, error = function(e) NA_real_) else NA_real_
n_subjects_all <- length(per_subject)
checks_df <- data.frame(
check = c("Enough subjects", "Enough raters", "Every subject rated by every rater", "No rater systematically higher or lower", "Residuals roughly normal"),
statistic = c(paste0(n, " complete subjects"), paste0(k, " raters"),
paste0(length(incomplete), " of ", n_subjects_all, " subjects incomplete"),
"rater effect in the two-way analysis of variance", "Shapiro-Wilk on the two-way residuals"),
p_value = c("", "", "", fmt_p(p_rater), fmt_p(shp)),
verdict = c(if (n >= 30) "holds" else if (n >= 15) "strained" else "violated",
if (k >= 3) "holds" else "strained",
if (!length(incomplete)) "holds" else if (length(incomplete) / n_subjects_all < 0.1) "strained" else "violated",
verdict_p(p_rater), verdict_p(shp, 0.01, 0.001)),
note = c("fewer than about 30 subjects leaves every ICC with a wide interval",
"with two raters the agreement and consistency forms are estimated from very little rater information",
"the two-way forms use only subjects every rater scored; excluded subjects can bias the result",
"a real rater effect lowers agreement ICC(2) below consistency ICC(3): report the form that matches how the scores will be used",
"the intervals assume roughly normal errors; heavy tails make them too narrow"),
stringsAsFactors = FALSE)
#' ## Method, assumptions, answer
excluded <- c(if (n_bad > 0) sprintf("%d row%s with a blank %s, %s or %s", n_bad, if (n_bad > 1) "s" else "", subject_name, rater_name, score_name),
if (n_dup > 0) sprintf("%d repeated rating%s of the same %s by the same %s averaged into one", n_dup, if (n_dup > 1) "s" else "", subject_name, rater_name),
if (length(incomplete)) sprintf("%d %s value%s not rated by every %s (%s%s)", length(incomplete), subject_name, if (length(incomplete) > 1) "s" else "", rater_name,
paste(utils::head(incomplete, 6), collapse = ", "), if (length(incomplete) > 6) ", ..." else ""))
method <- paste0(
"Intraclass correlation from two-way analysis of variance mean squares (Shrout and Fleiss 1979; intervals for the agreement forms by McGraw and Wong 1996) on ",
score_name, ": ", n, " ", subject_name, " values each rated by all ", k, " ", rater_name, " values (", n_used, " ratings used of ", n_in, " rows)",
if (length(excluded)) paste0("; excluded: ", paste(excluded, collapse = "; ")) else "",
if (length(ignored_cols)) paste0("; not used: ", paste(utils::head(ignored_cols, 12), collapse = ", "), " (reliability needs only the subject, rater and score)") else "",
"; 95% intervals from the F distribution; bands by Koo and Li (2016): below 0.5 poor, 0.5 to 0.75 moderate, 0.75 to 0.9 good, above 0.9 excellent",
"; the rater effect tested by the rater mean square against the residual mean square.")
assumptions <- list(
"Choose the form by design, not by size: agreement ICC(2) when raters are a sample and values must match, consistency ICC(3) when only these raters matter and only ordering must agree, and the average-of-raters forms only when the averaged score is what will be used.",
"Subjects are independent and represent the range the ratings will be used on; a narrow range of subjects lowers every ICC.",
"Errors are roughly normal with the same spread across subjects; the intervals are too narrow when they are not.",
"Only subjects rated by every rater enter the two-way forms.",
"An ICC says how well raters distinguish subjects; it does not say whether the scale measures the right thing.")
answer <- list(icc_agreement_single = round(icc21, 3), icc_consistency_single = round(icc31, 3), icc_agreement_average = round(icc2k, 3),
band_agreement_single = band(icc21), agreement_low = round(ci21[1], 3), agreement_high = round(ci21[2], 3),
rater_effect_p = if (p_rater < 1e-4) "<0.0001" else signif(p_rater, 3), subjects = n, raters = k,
most_lenient = raters[which.max(cm_)], most_severe = raters[which.min(cm_)], n = n_used)
results <- list()
#' The verdict and the headline are NOT places of a library tool (LAT-3130): the last mile writes them.
results$summary_metrics <- place_metric(list(icc_agreement = round(icc21, 3), icc_consistency = round(icc31, 3),
icc_average = round(icc2k, 3), subjects = n, raters = k, ratings = n_used), lead = "icc_agreement", place = "summary_metrics")
results$icc_interval <- place_interval(interval_df, term = "form", value = "estimate", low = "low", high = "high", place = "icc_interval")
results$variance_sources <- place_comparison(var_df, category = "source", value = "share_pct", place = "variance_sources")
results$rater_offsets <- place_comparison(rater_df, category = "rater", value = "offset", low = "ci_low", high = "ci_high", place = "rater_offsets")
results$rating_agreement <- place_relationship(long, x = "subject_mean", y = "score", series = "rater", place = "rating_agreement")
results$icc_table <- place_table(table_df, place = "icc_table")
results$widest_disagreement <- place_table(disagree_df, place = "widest_disagreement")
results$assumption_checks <- place_table(checks_df, place = "assumption_checks")
results$reliability_method <- list(kind = "metric", values = list(
method = method, n_in = n_in, n_used = n_used, excluded = as.list(excluded), assumptions = assumptions,
# the method card prints x_column -> y_column: what was read, and what it produced
x_column = paste(subject_name, rater_name, score_name, sep = ", "), y_column = sprintf("%d raters of %d subjects", k, n)),
value_order = list("n_used", "n_in"))
objects <- list() # filled by the object layer, not here
list(answer = answer, method = method, n = n_used, results = results, objects = objects,
json_output = list(answer = answer, method = method, n = n_used))
}