Which multiple-comparison correction do you actually need?
The reproducible worked example behind our multiple-comparisons lesson: 66 pairwise correlations on 1,599 wines, corrected four ways, cross-checked against R's own p.adjust.
The data
UCI Wine Quality (red), 1,599 wines, 12 chemistry variables, all 66 pairwise Pearson correlations. Real data, fetched from the UCI archive. No simulation.
The headline numbers
- Significant, uncorrected55
- Bonferroni43
- Holm43
- Benjamini-Hochberg54
- Expected by chance3.3
Swapping Bonferroni for Holm changed nothing here: both keep 43. The choice that moved the answer was family-wise error rate against false discovery rate, which moved 11 findings of 66.
Download and re-run it
Every figure in the document is derived when it is knitted, not typed in. Re-run the source and you should get the same numbers, or we have a problem worth hearing about.
- holm.html · The knitted document1.1 MB
- holm.Rmd · R Markdown source7 KB
- wine_correlation_pvalues.csv · The 66 p-values3 KB
- holm_example.py · Generates the data from UCI3 KB
What the review caught, and it was ours
The first draft of this document claimed the smallest p-values underflowed to zero in double precision. That was false: the smallest is 4.06e-220, and nothing underflowed. The claim had been imported from a note describing a different, earlier computation and repeated without testing it. It is corrected in the document, which now derives the real numbers live.
The honest limit
The 66 tests are not independent, because wine chemistry variables correlate. Bonferroni and Holm are valid under arbitrary dependence. Benjamini-Hochberg as used assumes independence or positive regression dependence, and the document says so.