Worked example · McNemar's test
McNemar's test or chi-square? Paired yes/no data, decided.
The reproducible worked example behind our McNemar lesson: 200 paired approve/deny decisions, agreement, kappa, and the four variants of the test.
The data
200 paired approve/deny decisions from two reviewers on the same cases, generated at seed 42 so it reproduces exactly.
The headline numbers
- Cases200
- Agreed170 (85%)
- Disagreed30
- Split24 / 6
- Cohen's kappa0.661
85% agreement sounds like reassurance. It is not. The 30 disagreements split 24 to 6, which is one reviewer holding a lower bar rather than two people being human.
Download and re-run it
Every figure in the document is derived when it is knitted, not typed in. Re-run the source and you should get the same numbers, or we have a problem worth hearing about.
- mcnemar.html · The knitted document1.2 MB
- mcnemar.Rmd · R Markdown source10 KB
- cases.csv · The 200 paired decisions5 KB
- mcnemar_example.py · Generates the data, seed 421 KB
The honest limit
The test tells you the two judges differ. It cannot tell you which one is correct. The remedy is calibration, not blame.