A publisher tested 12 headlines. Did any beat the first?

Upworthy, a publisher, showed 12 headlines for one story to readers and counted who clicked. We compared each of the 11 later headlines with the first one written. At the usual 0.05 bar, 5 of the 11 looked different. By chance alone, 11 tests at that bar would throw up about 0.55 false alarms. Once the bar is raised to account for all 11 tests, no headline clears it. Upworthy's file marked Headline 3 as its pick: 4.11% clicked against 3.89% for the first, with a p-value of 0.78.

Running many tests at once is called the multiple comparisons problem: each test is allowed a small chance of a false alarm, and the chances add up across the family of tests. A correction raises the bar for each result so the family as a whole stays honest.

Our position: when you compare many variants, set the bar for the whole batch, not for each result. A single test at 0.05 is a fair bet. Eleven of them are not.

The five at the top are the five that looked different at 0.05, raw. The p-value is the chance of a gap this big if the headlines were really the same; a smaller number looks stronger. After Holm's correction the smallest is 0.154. Nothing falls under 0.05. The other five headlines (5, 9, 10, 11, 12) were not different even before correcting.

Data: J. Nathan Matias, Kevin Munger, Marianne Aubin Le Quere and Charles Ebersole (2021), The Upworthy Research Archive, osf.io/jd64p, licensed CC BY 4.0. Changes: charts by MCP Analytics, computed from this data; each headline compared with the first. One test of 12 headlines, about 1,930 to 2,081 readers each. The first-created headline had 79 clicks from 2,032 impressions. The file marks no original, so we used the first created, a rule we wrote down before running anything.

What this means for your decisions

• Do not ship the headline that "won" a batch. Pick the best-looking of 12 and you may have picked the one that luck favoured most, as well as the one that may be good.
• Read the direction. All five that looked different were lower than the first headline, 2.5% to 2.7% against 3.89%. Only one headline of 11 beat the first at all (Headline 3, 4.11%, p-value 0.78: nothing there).
• A thin reference can fool you. Every comparison here leans on one headline with 79 clicks. If that count ran high by luck, every other headline looks worse. These data cannot tell us whether it did. That is the case for more readers, or a repeat, before acting.

Which of the three corrections to use depends on what a wrong call costs you.

• Bonferroni is the strictest: divide 0.05 by the number of tests. Here the bar is 0.00455 (0.05 / 11) and no headline reaches it.
• Holm is never less powerful than Bonferroni and holds the same guarantee. It kept none of the 5.
• Benjamini-Hochberg accepts a set share of false discoveries, 5%, among the results it keeps. It is the one to reach for when you screen many options and can live with a few wrong ones. It kept none either: its smallest adjusted value was 0.057, just over the line.

How to check your own batch of tests

1. Count the family before you look. Every headline, segment or metric you compare in one go is part of it.
2. Name the reference first. Say which one the others are compared with, and write it down before the data comes in. We did.
3. Run the corrected comparison. Raw p-values in, adjusted p-values out. Act on the adjusted ones.
4. Treat what survives as a lead, not a verdict. A result that clears the bar after correction is still worth a repeat on fresh readers before it moves a budget.

Run it on your own tests

Upload one column with a label for each comparison and one with its p-value. The free multiple comparisons test puts Bonferroni, Holm and Benjamini-Hochberg side by side and shows which results survive each.

Open the free multiple comparisons test

For why corrections work and which to choose, read The Multiple Comparisons Problem.

Questions business owners ask

Is 0.05 the wrong bar?

It is a fair bar for one test. The trouble starts when you use it eleven times. Here 11 tests at 0.05 would be expected to give about 0.55 false alarms on luck alone.

Why did correcting remove all five?

The correction asks each result to clear a higher bar. The best raw p-value was 0.014. Holm turned it into 0.154. A gap that looks decisive on its own can look ordinary once you admit you also looked at ten others.

Does "not significant" mean the headlines are the same?

No. It means these readers did not give us enough to tell them apart from luck. Larger tests may.

Doesn't Upworthy's own testing tool pick a winner?

It does, and its file flags one: Headline 3, the only headline above the first. We did not use that flag. Against the first headline, Headline 3 is 4.11% to 3.89% with a p-value of 0.78.

How many comparisons is too many?

There is no fixed count. The question is whether you would have acted on any one of them. If yes, they are one family.

Should I use Holm or Benjamini-Hochberg?

Use Holm when one wrong call is expensive. Use Benjamini-Hochberg when you are screening many options and a few false leads are cheap to drop.

The method in full

Family: the 11 later headlines of one Upworthy test (clickability_test_id 5308c3271e030e5930000003), each compared with the earliest-created headline on the share who clicked (clicks out of impressions). Test: two-sided two-proportion test with continuity correction (R prop.test), alpha 0.05. Corrections: Bonferroni, Holm and Benjamini-Hochberg through base R p.adjust. Counts at 0.05: 5 of 11 raw; 0 kept by Bonferroni, Holm and Benjamini-Hochberg. Bonferroni threshold 0.00455. Smallest p-value 0.01398 (Headline 4), Holm-adjusted 0.1538, Benjamini-Hochberg-adjusted 0.0574. Upworthy's own significance, first_place and winner columns are shown beside the result in the key and never used in it. The rules were committed before the analysis script. The free multiple comparisons test runs our standard_multiple_comparisons_v2 tool. For the estimators and limits in detail, read The Multiple Comparisons Problem.