Incrementality: Did the Offer Cause Anything, or Just the Letter?

A match offer in a fundraising letter was followed by 736 people giving. About 140 of them gave because of the offer.

The other 596 would have come anyway. We know because the same charity sent a letter without the offer to a randomly chosen group of past donors, and about that many of them gave too. More people gave; the extra money is likely but not certain: about $154 per 1,000 letters, 95% range -$3 to $311.

That gap, between the giving an action gets credit for and the giving it caused, is what incrementality measures. Here is a real test, the one formula you need, and how to run the same test on your own list.

Also on this topic: The idea, no data and The analyst version.

Our position: judge an offer on the number it caused, not the number that followed it. Hold out a random group that gets everything except the thing you are testing, and compare.

The answer first: a real fundraising test

In a study mailed in August 2005 and published in 2007, Dean Karlan and John List took 50,083 past donors to a charity and split them at random. Two thirds got a letter that offered to match their gift. One third got the same letter without the offer. Before the mailing, the two groups looked the same: the same months since their last gift, the same number of past gifts, about the same highest past gift. Then the charity counted who gave in response to the letter.

The offer worked on giving: more people gave than chance would explain. On money, the picture is less sure. The 95% range runs from a small loss to a clear gain, so the extra money is likely but not certain.

One thing about this test is easy to miss. The group without the offer still got a letter. So the test measures what the offer caused, not what mailing caused. It cannot tell you how many people would have given with no letter at all.

The offer worked on giving: more people gave than chance would explain. On money, the picture is less sure. The 95% range runs from a small loss to a clear gain, so the extra money is likely but not certain.

One thing about this test is easy to miss. The group without the offer still got a letter. So the test measures what the offer caused, not what mailing caused. It cannot tell you how many people would have given with no letter at all.

*Data: Dean Karlan and John A. List (2007), Does Price Matter in Charitable Giving?, Harvard Dataverse, doi:10.7910/DVN/27853, CC0. The figures above are our computation from the file; the data are unchanged.*

What this means for your budget

• Credit and cause are different numbers. The offer letter was followed by 736 people giving. Credit all of them to the offer and you overstate its effect by about five times. About 140 is the number to plan on.
• A range is part of the answer. The 95% range for the 140 is 52 to 223. Say the range to whoever signs off the budget.
• Extra giving does not mean extra money. More people gave, but the 95% range for the money runs from a $3 loss to a $311 gain per 1,000 letters, so the extra money is likely but not certain. If the offer costs you something, such as a fund that matches gifts, compare that cost with the middle figure and with the low end.
• One test answers one question. This one asks whether the offer beat the plain letter. It does not ask whether the letter beat nothing.

What incrementality is

Incrementality is the part of a result an action caused, beyond what would have happened anyway.

You measure it by comparing a group that got the action with a group that did not, chosen at random:

• Lift = result in the action group minus result in the comparison group
• Relative lift = lift divided by the comparison group's result

In the test above, 2.204% of the offer group gave and 1.786% of the other group gave. The lift is +0.418 points, about 0.42. Divided by 1.786%, the relative lift is +23%: for every four people who would have given anyway, about one more gave.

The comparison group is what makes this work. It shows you "anyway": the giving you would have had without the offer. Everything above that line is what the offer caused.

Incrementality vs attribution

Attribution and incrementality answer different questions.

• Attribution asks which action gets the credit. Everyone who gave after the offer letter counts: 736 people.
• Incrementality asks how many of those people the offer caused to give. Take away the 596 who would have given anyway, and about 140 are left.

Attribution is useful for seeing the path people take. It cannot tell you what would have happened without the action, because it only ever looks at people who got it. When you decide whether to keep an offer, the number you need is the 140, not the 736.

Two other comparisons are often mistaken for incrementality:

• Before vs after. Giving the week after a mailing against the week before. The week after can be higher for reasons that have nothing to do with the letter: a season, a holiday, a news story. The random comparison group lives through the same week, so it cancels those out.
• Price elasticity. How much demand moves when the price moves. That is a different question; see the price elasticity guide.

How to run one on your own list

1. Pick the action and the result. One thing to test, one number to judge it by: whether someone gave or bought, how much they spent, whether they visited.
2. Hold out a random group before you send. Take your list and set part of it aside at random, by a coin flip or a random number, not by region, date or who looks least likely to respond. The randomness is what lets you say "caused". In the charity's test, the comparison group was a third of the list.
3. Give the held-out group everything except the thing you are testing. If you are testing an offer, they get the same message without it. If you are testing the message itself, they get nothing, and then you learn what the message caused.
4. Wait for the window to close. Decide it in advance and use the same window for both groups.
5. Compare the two groups. The difference is the lift; the lift divided by the comparison group's result is the relative lift. To turn it into a count, multiply the lift by the size of the group that got the action.

How big should the holdout be? It depends on how rare the result is and how big a lift you need to see. Giving here is rare: 1.79% without the offer. To pick up a lift the size of this one (0.42 points) with the usual confidence (80% power, 5% significance), this test needed about 17,561 people in each group. The held-out group here had 16,687, a little under that, and the larger offer group of 33,396 made up part of the difference; the tool marks that check strained, so read the ranges with that in mind. Your list will need its own number, from your own rate and the lift you want to see. The smaller the lift you are looking for, and the rarer the result, the bigger both groups have to be.

What you need and what you get back

What your file needs:

• One row per person.
• Which group they were in: got the action, or held out.
• The result for each person: gave or not, bought or not, or how much.

What you get back (from the free randomized incrementality v2 tool):

• The result in each group, with its 95% range.
• The difference between the group that got the action and the comparison group, and the relative change, each with its range.
• Whether the held-out group is big enough to be a real comparison.
• How many people the test needed, and the smallest difference it could detect.
• Each condition the result relies on, and whether it holds.
• The method named and the code that ran it.

Try it: the free randomized incrementality v2 tool runs this comparison on your own file. Files over 10,000 rows are sampled on the free page, so its figures will be close but not exact.

Questions business owners ask

Is a holdout group worth the lost gifts or sales?

The holdout is how you find out whether those gifts or sales were lost at all. In the charity's test, about 596 of the people who got the offer would have given anyway. The holdout told us that.

What if I do not have enough conversions to run a test?

Rare results need big groups. At a giving rate under 2%, this test needed about 17,561 people per group to see a lift of 0.4 points. If you cannot reach that, you have three honest options: make the groups bigger, wait longer so more people have the chance to respond, or test something you expect to move the result by more. Judging on an easier result, such as opens or clicks, gives a number you can reach but answers a smaller question.

In which situation should I use incrementality testing?

When you can choose who gets the action, the action costs real money or effort, and you plan to repeat it. A promotion, a message, an offer, a mailing or an ad you can switch off for a random group are all good fits. If you cannot hold anyone out, see the next question.

Can I measure incrementality without a holdout group?

Not with the same confidence. Without a random group that got nothing, you are estimating "anyway" from something else, such as last month or another region, and the estimate carries everything else that changed. When a holdout is impossible, methods such as causal impact and synthetic control build a stand-in, and say how sure they are.

How do I show incremental revenue to leadership?

Show three things: the comparison group, so they can see "anyway"; the number caused, with its range; and the money per 1,000 letters, with its range. Here that is about 140 people caused to give (95% range 52 to 223) and about $154 per 1,000 letters (95% range -$3 to $311). Say plainly what is likely and what is not certain. A range that reaches below zero is an honest answer, and it tells them how much to trust the middle.

Is a study from 2005 still relevant?

The method has not changed: hold out a random group, compare. The figures belong to that charity and that mailing. Yours will differ, which is the reason to run your own.