Executive Summary
Is the difference between reviewer a and reviewer b systematic? Verdict: real.
The two reviewers show a statistically significant systematic difference: reviewer b approves at 72% versus reviewer a at 63%, a gap of 9 percentage points on identical cases. McNemar's test (uncorrected chi-square 10.8, p = 0.00102) confirms this is unlikely chance noise. The split of disagreements—24 cases where only reviewer b approves versus 6 where only reviewer a does—is too lopsided to be random. This is a calibration issue, not a competence one: the sources broadly agree (85%) but hold different approval standards. Next step: align the approval threshold between reviewers or document the intentional difference in mandate.
Analysis Overview
McNemar's test on 200 paired cases: reviewer a vs reviewer b.
Of 200 paired cases, reviewers agree on 170 (85%), yet the 30 disagreements show a clear pattern: reviewer b says 'approve' on 24 cases where reviewer a does not, versus only 6 the reverse. This 4:1 imbalance in the discordant pairs is what the test detects—not a failure of agreement, but a systematic shift in approval threshold. The two sources rank cases similarly (kappa 0.66) but hold different bars. Next step: examine the 30 disagreement cases to identify whether the threshold gap is defensible or signals drift in reviewer b's calibration.
Data Quality
200 paired cases used; verdict values 'approve' and 'deny'.
All 200 rows remained intact with no exclusions. Both reviewers used identical verdict categories ('approve' and 'deny'), making the pairing valid. The effective sample for McNemar is the 30 discordant pairs; the 170 concordant cases do not carry signal for this test. This design—paired judgments on the same cases—rules out the ordinary chi-square test of independence and makes McNemar the appropriate choice, avoiding false significance from the non-independence of the two verdicts on a single case.
McNemar Results: Name Your Variant
All four test variants on the same table, headline = uncorrected.
| Variant | Statistic | P Value |
|---|---|---|
| Uncorrected chi-square (headline) | 10.8 | 0.001 |
| Continuity-corrected (Edwards 1948; R's default) | 9.633 | 0.0019 |
| Exact binomial | — | 0.0014 |
| Mid-p (Lancaster 1961) | — | 9.00e-04 |
Four test variants on the same 2×2 table yield p-values ranging from 0.001 to 0.0019. The uncorrected chi-square (10.8, p = 0.00102) is the headline statistic, consistent with recent methodological guidance (Fagerland, Lydersen & Laake 2013) recommending against the Edwards continuity correction. The mid-p variant (p = 0.0009) and exact binomial (p = 0.0014) cluster tightly around the uncorrected value, all well below conventional significance thresholds. The difference between variants is small enough that the conclusion—a real systematic difference—holds across all four, though the exact p-value depends on which variant is cited. Specify your variant in any external report to enable reproducibility.
The Disagreements Carry the Evidence
30 discordant cases, split 6 vs 24.
Of the 30 discordant pairs, reviewer b says 'approve' on 24 cases where reviewer a does not, while reviewer a says 'approve' on only 6 cases where reviewer b does not. Under the null hypothesis of no systematic difference, these should split near 50/50; the observed 24:6 ratio is far too lopsided to be chance (McNemar p = 0.00102). This directional imbalance—not the mere existence of disagreement—is what the test detects. The 170 concordant cases never enter the calculation; they establish context but do not influence the test statistic.
Marginal Rates: The Effect Size
'approve' rate: reviewer a 63% vs reviewer b 72%.
The short answer
Reviewer B approves cases at a materially higher rate than Reviewer A: 72% versus 63%, a 9 percentage-point gap on identical cases. This gap is real enough to shape how you'd allocate review work or set thresholds, but it is not enormous — both reviewers are operating in a similar approval range, not at opposite extremes.
The detail
Reviewer A approves 0.63 (63%) of cases; Reviewer B approves 0.72 (72%). The difference of 0.09 (9 percentage points) represents a directional and statistically significant pattern (McNemar test confirms the gap is not noise). The substantive question is whether 9 percentage points matters operationally — for example, whether it suggests different risk tolerance, case-selection bias, or genuine disagreement about borderline cases.
What this can't tell you
These rates describe approval propensity on the sample reviewed but do not reveal whether the gap reflects differences in reviewer judgment quality, case difficulty, or workload patterns. A deeper analysis of case-level agreement (which cases each reviewer splits on) would clarify whether the 9-point gap is driven by systematic disagreement on a subset or scattered across the full caseload.
Agreement vs McNemar: Different Questions
Agreement 85%, kappa 0.66, and the test still rejects.
| Cell | Count | Enters Test |
|---|---|---|
| both 'approve' | 120 | no |
| both 'deny' | 50 | no |
| only reviewer a 'approve' | 6 | yes (b) |
| only reviewer b 'approve' | 24 | yes (c) |
The full paired table reveals why high agreement (85%) and a significant McNemar test coexist: they measure different cells. The 120 cases both approve and 50 both deny (170 total concordant) feed agreement statistics (kappa 0.66). The 6 + 24 = 30 discordant cases feed McNemar. Agreement answers 'how often do they match?'; McNemar answers 'when they differ, does it lean one way?' Both are true simultaneously. Reviewer b's systematic tendency to approve more often emerges not from disagreement frequency but from the direction of disagreement. Reporting both measures provides a complete picture: the sources are broadly aligned but systematically offset.
McNemar Test — Paired Proportions
When the same cases are judged twice (two reviewers, two diagnostic tests, before vs after on the same subjects), does one judging say yes more often? McNemar's test reads ONLY the disagreements and asks whether they lean one way — which is why two sources can agree on most cases and still be systematically shifted where they differ.
Why This Method?
The paired design makes the ordinary chi-square test of independence illegal (the two verdicts on a case are not independent samples). McNemar is the standard paired-proportions test; all four common variants are computed and NAMED, because software defaults diverge and the p-values differ on identical data.
What This Analysis Covers
- The paired 2x2 table with the cells that enter the test marked
- Uncorrected (headline, per Fagerland 2013), continuity-corrected,
exact binomial, and mid-p variants
- Direction and effect size via the marginal positive rates
- Agreement + kappa alongside, as explicit contrast (different question)
- Low-power caution when discordant pairs < 10
Standard Library
Platform standard-library module (LAT-2253): runs on ANY dataset via the semantic mapping {case, verdict_a, verdict_b}. All narrative is derived from the user's own column names and computed values.
suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))Core Analysis Pipeline
compute_shared <- function(df, params, col_map = list()) {
# === SHARED EXPORTS ===
# initial_rows/final_rows/rows_removed $ row accounting (cases)
# case_h / va_h / vb_h $ humanized names of the three mapped columns
# n_cases $ paired cases used
# pos_level / neg_level $ which verdict value is counted as "positive"
# both_pos / both_neg / b_only_a / c_only_b $ the 2x2 cells (b = A-only positive, c = B-only positive)
# m $ discordant pairs (b + c)
# agreement / kappa $ raw agreement and Cohen's kappa (context, NOT the test)
# rate_a / rate_b / rate_gap_pp $ marginal positive rates + gap in percentage points
# chi2 / p_chi2 $ McNemar uncorrected (THE HEADLINE, per Fagerland 2013)
# chi2_cc / p_cc $ Edwards continuity-corrected (shown for reconciliation with R's default)
# exact_p / mid_p $ exact binomial and mid-p (Lancaster 1961)
# low_power $ TRUE when discordant pairs < 10 (Pembury Smith & Ruxton 2020)
# table_df / variants_df / split_df / rates_df
# metrics / json_output
# === /SHARED EXPORTS ===
case_h <- humanize_semantic("case", col_map)
va_h <- humanize_semantic("verdict_a", col_map)
vb_h <- humanize_semantic("verdict_b", col_map)Step 1: Validate the mapped columns
initial_rows <- nrow(df)
for (req in c("case", "verdict_a", "verdict_b")) {
if (is.null(df[[req]])) {
stop(sprintf("column_mapping must map the '%s' column (%s / %s / %s).",
req, case_h, va_h, vb_h))
}
}Step 2: Clean the two verdict columns; drop rows missing either verdict
df$case <- trimws(as.character(df$case))
df$verdict_a <- trimws(as.character(df$verdict_a))
df$verdict_b <- trimws(as.character(df$verdict_b))
keep <- !is.na(df$verdict_a) & df$verdict_a != "" &
!is.na(df$verdict_b) & df$verdict_b != "" &
!is.na(df$case) & df$case != ""
n_invalid <- sum(!keep)
df <- df[keep, , drop = FALSE]
if (nrow(df) < 10) {
stop(sprintf("Only %d usable cases after removing rows with a missing %s, %s, or %s; need at least 10.",
nrow(df), case_h, va_h, vb_h))
}Step 3: The verdicts must be dichotomous — exactly two levels across both columns
levs <- sort(unique(c(df$verdict_a, df$verdict_b)))
if (length(levs) != 2) {
stop(sprintf(
"McNemar's test needs exactly two verdict values; found %d (%s) across %s and %s. More than two categories needs the Stuart-Maxwell test; recode or filter first.",
length(levs), paste(utils::head(levs, 6), collapse = ", "), va_h, vb_h))
}Step 4: Which level counts as "positive"? A keyword match, a user
override (module_parameters$positive_level), or the first level sorted. The choice never changes the test (b and c just swap roles); it only anchors how direction is phrased, so it is always NAMED in the output.
pos_keywords <- c("yes", "y", "true", "1", "approve", "approved", "positive",
"pass", "passed", "success", "present", "detected")
pos_level <- params$positive_level %||% NULL
if (is.null(pos_level) || !(pos_level %in% levs)) {
kw_hit <- levs[tolower(levs) %in% pos_keywords]
pos_level <- if (length(kw_hit) >= 1) kw_hit[1] else levs[1]
}
neg_level <- setdiff(levs, pos_level)
final_rows <- nrow(df)
rows_removed <- initial_rows - final_rows
n <- final_rowsStep 5: The paired 2x2 table. Only b and c enter the statistic.
a_pos <- df$verdict_a == pos_level
b_pos <- df$verdict_b == pos_level
both_pos <- sum(a_pos & b_pos)
both_neg <- sum(!a_pos & !b_pos)
b_only_a <- sum(a_pos & !b_pos) # b: first source positive, second not
c_only_b <- sum(!a_pos & b_pos) # c: second source positive, first not
m <- b_only_a + c_only_bStep 6: Context statistics (agreement and kappa answer a DIFFERENT
question than the test; both are reported so the contrast is visible)
agreement <- (both_pos + both_neg) / n
rate_a <- (both_pos + b_only_a) / n
rate_b <- (both_pos + c_only_b) / n
pe <- rate_a * rate_b + (1 - rate_a) * (1 - rate_b)
kappa <- if (isTRUE(all.equal(pe, 1))) NA_real_ else (agreement - pe) / (1 - pe)
rate_gap_pp <- round(100 * (rate_b - rate_a))Step 7: McNemar variants. Headline = UNCORRECTED, per Fagerland,
Lydersen & Laake (2013); Edwards continuity-corrected shown only so a reader reconciling against R's default output knows why numbers differ; exact binomial and mid-p computed from the discordant pairs.
if (m > 0) {
chi2 <- (b_only_a - c_only_b)^2 / m
p_chi2 <- stats::pchisq(chi2, df = 1, lower.tail = FALSE)
chi2_cc <- (abs(b_only_a - c_only_b) - 1)^2 / m
chi2_cc <- max(0, chi2_cc)
p_cc <- stats::pchisq(chi2_cc, df = 1, lower.tail = FALSE)
kmin <- min(b_only_a, c_only_b)
exact_p <- min(1, 2 * stats::pbinom(kmin, m, 0.5))
mid_p <- max(0, exact_p - stats::dbinom(kmin, m, 0.5))
} else {
chi2 <- NA_real_; p_chi2 <- NA_real_; chi2_cc <- NA_real_; p_cc <- NA_real_
exact_p <- NA_real_; mid_p <- NA_real_
}
low_power <- m < 10
direction_txt <- if (m == 0) {
sprintf("the two sources never disagreed on any of the %d cases", n)
} else if (c_only_b > b_only_a) {
sprintf("%s says '%s' more often: it is positive on %d cases where %s is not, against %d the other way",
vb_h, pos_level, c_only_b, va_h, b_only_a)
} else if (b_only_a > c_only_b) {
sprintf("%s says '%s' more often: it is positive on %d cases where %s is not, against %d the other way",
va_h, pos_level, b_only_a, vb_h, c_only_b)
} else {
sprintf("the %d disagreements split evenly(%d vs %d): no directional lean", m, b_only_a, c_only_b)
}
verdict_txt <- if (m == 0) {
"no disagreements at all, so the marginal rates are identical by construction and the test does not apply"
} else if (is.na(p_chi2)) {
"the test could not be computed"
} else if (p_chi2 < 0.05) {
sprintf("the split is unlikely to be chance(uncorrected chi-square %.2f, p = %s): the two sources genuinely differ in how often they say '%s'",
chi2, format(signif(p_chi2, 3), scientific = FALSE), pos_level)
} else {
sprintf("the split is consistent with chance(uncorrected chi-square %.2f, p = %s): no evidence the two sources differ in their '%s' rate",
chi2, format(signif(p_chi2, 3), scientific = FALSE), pos_level)
}Step 8: Assemble display frames
table_df <- data.frame(
cell = c(sprintf("both '%s'", pos_level), sprintf("both '%s'", neg_level),
sprintf("only %s '%s'", va_h, pos_level), sprintf("only %s '%s'", vb_h, pos_level)),
count = c(both_pos, both_neg, b_only_a, c_only_b),
enters_test = c("no", "no", "yes(b)", "yes(c)"),
stringsAsFactors = FALSE
)
variants_df <- data.frame(
variant = c("Uncorrected chi-square(headline)", "Continuity-corrected(Edwards 1948; R's default)",
"Exact binomial", "Mid-p(Lancaster 1961)"),
statistic = c(round(chi2, 4), round(chi2_cc, 4), NA, NA),
p_value = signif(c(p_chi2, p_cc, exact_p, mid_p), 4),
stringsAsFactors = FALSE
)
split_df <- data.frame(
disagreement = c(sprintf("only %s says '%s'", va_h, pos_level),
sprintf("only %s says '%s'", vb_h, pos_level)),
count = c(b_only_a, c_only_b),
stringsAsFactors = FALSE
)
rates_df <- data.frame(
source = c(va_h, vb_h),
positive_rate = round(c(rate_a, rate_b), 3),
stringsAsFactors = FALSE
)
fmt_p <- function(p) if (is.na(p)) "n/a" else format(signif(p, 3), scientific = FALSE)
metrics <- list(
`Cases` = n,
`Discordant Pairs` = m,
`McNemar chi-sq` = if (is.na(chi2)) "n/a" else round(chi2, 2),
`p(uncorrected)` = fmt_p(p_chi2),
`Mid-p` = fmt_p(mid_p),
`Agreement` = paste0(round(100 * agreement), "%"),
`Kappa` = if (is.na(kappa)) "n/a" else round(kappa, 3)
)
json_output <- list(
answer = paste0(
"McNemar's test on ", format(n, big.mark = ","), " paired cases (",
va_h, " vs ", vb_h, ", counting '", pos_level, "' as positive): ",
"of ", m, " disagreements, ", direction_txt, "; ", verdict_txt, ". ",
"Variant note: this headline is the UNCORRECTED test per Fagerland et al. 2013 ",
"(mid-p = ", fmt_p(mid_p), "; R's default continuity-corrected p = ", fmt_p(p_cc), "). ",
"Context: raw agreement is ", round(100 * agreement), "% and kappa ",
if (is.na(kappa)) "n/a" else round(kappa, 2),
". Agreement and McNemar answer different questions(agreement reads the concordant ",
"cells; the test reads only the discordant ones), so high agreement and a significant ",
"test can coexist.",
if (low_power && m > 0) paste0(
" CAUTION: only ", m, " discordant pairs, below the ~10 recommended ",
"(power depends on discordant pairs, not total cases); treat as descriptive.") else ""
),
cards = lapply(
c("tldr", "overview", "preprocessing", "mcnemar_results",
"disagreement_split", "marginal_rates", "agreement_context"),
function(cid) list(id = cid, metrics = metrics)
)
)
list(
initial_rows = initial_rows, final_rows = final_rows,
rows_removed = rows_removed, n_invalid = n_invalid,
case_h = case_h, va_h = va_h, vb_h = vb_h,
n_cases = n, pos_level = pos_level, neg_level = neg_level,
both_pos = both_pos, both_neg = both_neg,
b_only_a = b_only_a, c_only_b = c_only_b, m = m,
agreement = agreement, kappa = kappa,
rate_a = rate_a, rate_b = rate_b, rate_gap_pp = rate_gap_pp,
chi2 = chi2, p_chi2 = p_chi2, chi2_cc = chi2_cc, p_cc = p_cc,
exact_p = exact_p, mid_p = mid_p, low_power = low_power,
direction_txt = direction_txt, verdict_txt = verdict_txt,
table_df = table_df, variants_df = variants_df,
split_df = split_df, rates_df = rates_df,
metrics = metrics, json_output = json_output
)
}