ds@mcpanalytics.aiEmail your data and your question and get an answer about your own data, with the method shown.Email your data and question. Get an answer.
Worked example · Multiple comparisons

Which multiple-comparison correction do you actually need?

By MCP Analytics Team | Published , updated

The reproducible worked example behind our multiple-comparisons lesson: 66 pairwise correlations on 1,599 wines, corrected four ways, cross-checked against R's own p.adjust.

The data

UCI Wine Quality (red), 1,599 wines, 12 chemistry variables, all 66 pairwise Pearson correlations. Real data, fetched from the UCI archive. No simulation.

To see which of your own results survive correction for multiple testing, try the free multiple comparisons tool.

The headline numbers

Swapping Bonferroni for Holm changed nothing here: both keep 43. The choice that moved the answer was family-wise error rate against false discovery rate, which moved 11 findings of 66.

Download and re-run it

Every figure in the document is derived when it is knitted, not typed in. Re-run the source and you should get the same numbers, or we have a problem worth hearing about.

What the review caught, and it was ours

The first draft of this document claimed the smallest p-values underflowed to zero in double precision. That was false: the smallest is 4.06e-220, and nothing underflowed. The claim had been imported from a note describing a different, earlier computation and repeated without testing it. It is corrected in the document, which now derives the real numbers live.

The honest limit

The 66 tests are not independent, because wine chemistry variables correlate. Bonferroni and Holm are valid under arbitrary dependence. Benjamini-Hochberg as used assumes independence or positive regression dependence, and the document says so.

The brief for an AI agent

If an AI assistant is choosing a method for you, it needs different things than you do: the discriminating condition, the questions this gets mistaken for, the failure modes that return a plausible number instead of an error, and figures it can cite rather than generate. We publish that as its own document, written to one bar: an agent asked whether to run this analysis should be able to answer from it alone, including saying no.

Read the agent brief

Where this came from

Your turn

Bring your own data and the question you actually need answered.

MCP AnalyticsData science by email Send us your data and your question, and we will reply. ds@mcpanalytics.ai