McNemar's test or chi-square? Paired yes/no data, decided.
By MCP Analytics Team | Published , updated
The reproducible worked example behind our McNemar lesson: 200 paired approve/deny decisions, agreement, kappa, and the four variants of the test.
The data
200 paired approve/deny decisions from two reviewers on the same cases, generated at seed 42 so it reproduces exactly.
To test whether the same subjects changed between two of your own measurements, try the free paired comparison tool.
The headline numbers
- Cases200
- Agreed170 (85%)
- Disagreed30
- Split24 / 6
- Cohen's kappa0.661
85% agreement sounds like reassurance. It is not. The 30 disagreements split 24 to 6, which is one reviewer holding a lower bar rather than two people being human.
Download and re-run it
Every figure in the document is derived when it is knitted, not typed in. Re-run the source and you should get the same numbers, or we have a problem worth hearing about.
- mcnemar.html · The knitted document1.2 MB
- mcnemar.Rmd · R Markdown source10 KB
- cases.csv · The 200 paired decisions5 KB
- mcnemar_example.py · Generates the data, seed 421 KB
The honest limit
The test tells you the two judges differ. It cannot tell you which one is correct. The remedy is calibration, not blame.
The brief for an AI agent
If an AI assistant is choosing a method for you, it needs different things than you do: the discriminating condition, the questions this gets mistaken for, the failure modes that return a plausible number instead of an error, and figures it can cite rather than generate. We publish that as its own document, written to one bar: an agent asked whether to run this analysis should be able to answer from it alone, including saying no.
Where this came from
Your turn
Bring your own data and the question you actually need answered.
MCP AnalyticsData science by email Send us your data and your question, and we will reply. ds@mcpanalytics.ai