McNemar's Test: When Your A/B Test Has Paired Data

1. Introduction

1.1 Paired Binary Data

A team measures whether returning users convert before and after a redesign, using the same users in both periods. The two measurements on one user are not independent, so a test built for two separate groups answers the wrong question. McNemar (1947) proposed a test that works on the matched structure.

1.2 What This Paper Covers

• The matched-pairs 2 by 2 table and the test statistic
• Exact and approximate versions, with their error rates computed
• Reading direction and effect size
• Limits of before-and-after designs
• Extensions to more than two categories

2. Background

2.1 Why Not an Unpaired Test

The ordinary chi-square test and the two-proportion z-test assume the two samples are independent. With the same subjects in both, positively correlated outcomes make the variance of the difference smaller than an unpaired test assumes, so the unpaired test is usually conservative: it loses power. The numbers in section 4.1 show this. In other situations, for example matched pairs where the correlation is negative, the error can go the other way. Either way the p-value from an unpaired test does not describe the design.

2.2 The Matched-Pairs Table

Cross-tabulate the first measurement against the second for the same subjects:

The subjects counted in a and d did not change and say nothing about whether changes favor one direction. The test is about b and c.

3. Approach

3.1 The Test

If there is no systematic change, a discordant pair is equally likely to be of either kind, so b follows a binomial distribution with b + c trials and probability 0.5. The exact test is this binomial test. The usual approximation is

chi-square = (b - c)^2 / (b + c)        1 degree of freedom
with continuity correction: (|b - c| - 1)^2 / (b + c)

Swapping b and c leaves the statistic and the two-sided p-value unchanged, so direction is read from the counts themselves, not from the p-value.

3.2 Effect Size

The ratio c / b estimates the conditional (matched-pairs) odds ratio of becoming positive against becoming negative; the difference in the two marginal proportions is (c - b) / n. Report the counts, one of these measures and an interval, not only the p-value.

3.3 Assumptions

• Pairs are genuine: the same subject twice, or subjects deliberately matched.
• Different pairs are independent of each other.
• Both outcomes are binary and measured consistently at both times.

3.4 Before and After Is Not an Experiment

The test shows that changes were asymmetric. It cannot say why. Seasonality, learning, regression to the mean and outside events can all produce asymmetry between two periods. A concurrent control group, a longer baseline, or a staggered rollout is needed to attribute a change to an intervention.

3.5 More Than Two Categories

For a table with k categories measured twice, Bowker's test of symmetry asks whether the count moving from category i to j equals the count moving from j to i, for every pair. The Stuart-Maxwell test asks the different question of whether the two sets of marginal totals are equal. For three or more matched binary measurements on the same subjects, Cochran's Q test is the usual choice.

4. Worked Calculations

4.1 A Paired Before-and-After Example

A group of 1,000 users, 150 positive before and 180 after (15% to 18%), with the table below. The data are illustrative.

import numpy as np
from scipy import stats
from statsmodels.stats.contingency_tables import mcnemar
from statsmodels.stats.proportion import proportions_ztest

# Same 1,000 users measured before and after a change.  Rows: before (positive, negative); columns: after.
table = np.array([[135, 15],
                  [45, 805]])
b, c = table[0, 1], table[1, 0]
print(f"discordant pairs: positive to negative b = {b}, negative to positive c = {c}")
print(f"McNemar, asymptotic: chi2 = {(b - c) ** 2 / (b + c):.1f}, p = {mcnemar(table, exact=False, correction=False).pvalue:.5f}")
print(f"McNemar, exact binomial: p = {mcnemar(table, exact=True).pvalue:.5f}")

# The wrong analysis for comparison: treat the 150 of 1,000 'before' and 180 of 1,000 'after' as independent groups
z, p = proportions_ztest([150, 180], [1000, 1000])
print(f"two-proportion z-test that ignores the pairing: p = {p:.3f}")

discordant pairs: positive to negative b = 15, negative to positive c = 45
McNemar, asymptotic: chi2 = 15.0, p = 0.00011
McNemar, exact binomial: p = 0.00013
two-proportion z-test that ignores the pairing: p = 0.071

Only 60 of the 1,000 users changed state, and 45 of those 60 moved in the positive direction. The paired test detects that clearly. The unpaired test, which pretends the 150 and 180 come from separate groups, does not reach the 0.05 level on the same data.

4.2 How Often Does Each Version Reject When Nothing Has Changed?

Given a number of discordant pairs, the number in one direction is binomial when there is no effect, so each test's chance of a false rejection can be computed exactly:

import numpy as np
from scipy import stats

# Given m discordant pairs, under no effect b ~ Binomial(m, 0.5).  Exact probability that each test rejects at the 0.05 level:
print("discordant pairs   asymptotic (no correction)   asymptotic (continuity corrected)   exact binomial")
for m in (10, 20, 25, 50, 100):
    b = np.arange(m + 1)
    pmf = stats.binom.pmf(b, m, 0.5)
    chi2 = (b - (m - b)) ** 2 / m
    chi2_cc = np.maximum(np.abs(b - (m - b)) - 1, 0) ** 2 / m
    exact_p = np.minimum(1, 2 * stats.binom.cdf(np.minimum(b, m - b), m, 0.5))
    rej = lambda p: pmf[p < 0.05].sum()
    print(f"{m:16d}   {rej(stats.chi2.sf(chi2, 1)):26.3f}   {rej(stats.chi2.sf(chi2_cc, 1)):33.3f}   {rej(exact_p):14.3f}")

discordant pairs   asymptotic (no correction)   asymptotic (continuity corrected)   exact binomial
              10                        0.021                               0.021            0.021
              20                        0.041                               0.041            0.041
              25                        0.043                               0.043            0.043
              50                        0.065                               0.033            0.033
             100                        0.057                               0.035            0.035

The exact test never exceeds 0.05. The uncorrected approximation is conservative at some counts and above 0.05 at others (0.065 at 50 and 0.057 at 100 discordant pairs), because the counts are discrete. The continuity-corrected version tracked the exact test in this run. With a modest number of discordant pairs, the exact test costs little and removes the question.

4.3 More Than Two Categories

Engagement level (none, low, medium, high) for the same 500 users before and after a campaign, with illustrative counts:

import numpy as np
from statsmodels.stats.contingency_tables import SquareTable

# Engagement level (none, low, medium, high) for the same 500 users before (rows) and after (columns).  Illustrative counts.
engagement = np.array([[280, 45, 12, 3],
                       [38, 56, 18, 5],
                       [8, 15, 12, 2],
                       [2, 3, 1, 0]])
t = SquareTable(engagement)
sym, hom = t.symmetry(), t.homogeneity()
print(f"Bowker test of symmetry:           chi2 = {sym.statistic:.2f}, df = {int(sym.df)}, p = {sym.pvalue:.2f}")
print(f"Stuart-Maxwell marginal homogeneity: chi2 = {hom.statistic:.2f}, df = {int(hom.df)}, p = {hom.pvalue:.2f}")

Bowker test of symmetry:           chi2 = 2.70, df = 6, p = 0.85
Stuart-Maxwell marginal homogeneity: chi2 = 2.50, df = 3, p = 0.47

Neither test finds evidence that engagement moved systematically in this table. An omnibus test spreads its power over every pair of categories, so if you had a specific transition in mind in advance, test it directly by collapsing categories and applying McNemar's test.

5. Practical Guidance

6. Conclusion

McNemar's test is the standard way to compare two binary measurements on the same subjects. In the worked example it found a clear effect where an unpaired test did not, and the exact version was valid at every discordant count while the uncorrected approximation was not always. It shows asymmetry between two measurements, not the reason for it, so pair it with a design that rules out other explanations when cause matters.