An A/B test shows two versions to two groups and counts who clicked. Version A got 103 clicks from 5,867 views. Version B got 51 clicks from 5,868 views. B did about half as well. The question is whether that gap is real or just luck.

Luck is a poor explanation here. If the two pictures truly performed the same, a gap this large would show up about 2 times in 100,000. That is one test on one pair of pictures, and we could not rule out that the test was stopped early (see the limits below), so it is not a rule about thumbnails.

The question and the data

The data are from the Upworthy Research Archive, a public set of A/B tests run by a media site (CC BY 4.0). Every test shows one headline with different pictures or wording. We picked ONE test before looking at any click counts: the confirmatory test with the id 51436080220cb80002000970. The two versions use the same headline and different thumbnail pictures, so this is a thumbnail test.

Basis for every figure here: one test, 11,735 views, clicks counted per view.

How the test works

Think of two coins that may or may not be the same. You flip each many times and ask if the heads rates differ more than chance would allow.

Click rate for A is 1.76 percent (103 of 5,867). Click rate for B is 0.87 percent (51 of 5,868). The difference B minus A is minus 0.89 points, with a 95 percent interval of minus 1.31 to minus 0.48 points. B's rate is 50.5 percent lower than A's.

Three checks, one answer

  1. Fisher exact test: p = 0.0000221. It counts every possible split of the clicks and asks how rare ours is.
  2. Continuity-corrected test of two proportions: p = 0.0000350. This is the method the free tool uses.
  3. Odds ratio: B's odds of a click are 0.49 times A's, with a 95 percent interval of 0.34 to 0.69. The interval does not include 1, so it agrees.

Three methods, one story. The tool's p-value differs slightly from the exact one because they count differently, and that is expected.

Did the split hold up?

The groups came out almost even: 5,867 views for A and 5,868 for B. The sample ratio check gives p = 0.99, so nothing suggests a broken split.

How many views did it need?

At the size of gap we saw, about 2,587 views per version gives 80 percent power. This test had more than twice that. At 5,868 views per version, the smallest change from A's rate that the test could reliably detect was a rate of 2.50 percent, a difference of 0.74 points.

What this does and does not show

It shows B got fewer clicks than A and that luck is a poor explanation. It does not show why, and it does not say any thumbnail style wins elsewhere. One more limit: the archive's pages do not say how long each test ran or how its end was decided, so we cannot rule out early stopping. They do say unequal group sizes affected some tests, mostly from June 2013 to January 2014; this test ran in February 2013 and its groups came out almost even.

Try it on your own two counts with the free A/B test tool.

Data: Upworthy Research Archive, CC BY 4.0. Changes: one test picked before any click count was read.