Estimating Causal Lift from a Randomized Holdout
A randomized control arm turns a difference in rates into an effect, with its interval.
The short answer
A mailing promised to tell the neighbors whether you voted. In a random sample of 5,000 voters who got that mailing, 1,872 voted in the August 2006 Michigan primary. In a random sample of 5,000 who got no mailing, 1,490 did. The lift is 7.64 percentage points, 95% range 5.79 to 9.48, and about 382 of the 1,872 voters voted because of the mailing (95% range 290 to 474). The other 1,490 would have voted anyway.
The difference between two rates only has a causal reading when a random draw decided who was in which group. Here one did. That is what lets us say the mailing caused the extra votes, and not that it was followed by them.
Our position: estimate the lift as a difference in rates with an interval that fits small rates, report it as a count of cases caused with the same interval, and size the control arm before you collect, never after. The free tool does the estimate, the interval and the check on your file.
The design: a random control arm
Take one list. Before the treatment, assign each unit at random to a treated arm or to a control arm that gets nothing. Measure the same outcome in every arm over the same window, fixed in advance.
Randomization is what makes the comparison causal. Because assignment is independent of everything about the unit, the arms are alike in expectation on every characteristic, measured or not, so a difference in outcomes can be attributed to the treatment. In this experiment the treatment is a mailing promising to tell the neighbors whether the voter voted. The control is no mailing.
Three terms are used throughout:
- Credited: every unit in the treated arm that shows the outcome. This is what an attribution report counts. Here, 1,872 voters.
- Would have shown it anyway: the control arm's rate times the treated arm's size. It is the estimate of the treated arm's outcome in the world where nobody was treated. Here, 1,490.
- Caused: credited minus would-have-shown-it-anyway. Here, about 382.
The data behind it
Data: Alan S. Gerber, Donald P. Green and Christopher W. Larimer (2008), Social Pressure and Voter Turnout, Yale ISPS, doi:10.60600/YU/CGMWNW, CC0 1.0. The figures here are our computation from the file; the data are unchanged. Only two columns are used: treatment and voted.
The experiment mailed households in Michigan before the August 2006 primary. We drew a fixed random sample (seed 2008) of 5,000 voters from the Control group and 5,000 from the Neighbors group, 10,000 rows in all.
The experiment had five arms: Control, which got no mailing, and four mailings named Civic Duty, Hawthorne, Self and Neighbors. On the full file of 344,084 voters, turnout was 29.66% for Control, 31.45% for Civic Duty, 32.24% for Hawthorne, 34.52% for Self and 37.79% for Neighbors. Neighbors is the strongest mailing and the one this article follows. The chart under the short answer shows all five with their 95% ranges. Those ranges treat each voter as independent, and because the mailings went to households they are slightly wider than shown.
| Arm | Voters | Voted | Rate |
|---|---|---|---|
| Neighbors mailing | 5,000 | 1,872 | 37.44% |
| Control, no mailing | 5,000 | 1,490 | 29.80% |
On the full file, all 229,444 voters in these two arms, the lift is 8.1 points, which sits inside the sample's 95% range of 5.79 to 9.48.
The estimand
For a treated arm *k* and the control *c*, with rates p_k and p_c:
- Lift = p_k − p_c, in percentage points. This is the average effect of being assigned to the treatment on the rate. Here 37.44 − 29.80 = 7.64 points.
- Relative lift = p_k / p_c − 1. Here about 26 percent. People usually quote this one; report the lift in points beside it.
- Cases caused = lift × n_k, where n_k is the size of the treated arm. Its interval is the lift interval times n_k. Here about 382, 95% range 290 to 474.
Note what this is. It is the effect of being *assigned* to the treatment (intent to treat), not the effect on people who read the mailing. The limits section returns to this.
The estimators, as the tool computes them
A rate per arm: Wilson interval. Each arm's rate gets a 95% Wilson score interval, which behaves well for small rates and small counts where the textbook normal interval can fall below zero.
The lift: Newcombe interval. The difference of two independent proportions gets a Newcombe interval, built from the two Wilson intervals (the hybrid score method). It keeps close to its stated coverage for small rates and unequal arms. Here: 5.79 to 9.48 points.
The test: two-proportion z-test. Each treated arm is tested against the control with the two-proportion z-test, without continuity correction. Here p is below 0.0001.
Several treated arms: Holm adjustment. When more than one arm is compared with the same control, the p-values share one Holm adjustment, which controls the chance of at least one false positive across the family and never keeps fewer results than Bonferroni. With one treated arm, as here, it changes nothing.
An amount: Welch interval. If an amount column is supplied, the difference in mean amount per unit gets a Welch interval, which does not assume equal variances. This experiment has a yes or no outcome only, so no amount is reported.
The worked result and its range
| Neighbors mailing | |
|---|---|
| Voters in the sample | 5,000 |
| Voted | 1,872 (37.44%) |
| Control voted | 1,490 of 5,000 (29.80%) |
| Lift | 7.64 points (95% range 5.79 to 9.48), p below 0.0001 |
| Relative lift | about 26% |
| Would have voted anyway | 1,490 |
| Caused | about 382 (95% range 290 to 474) |
| Caused as a share of the voters who voted | about 20% |
An attribution report would have credited the mailing with all 1,872 voters. The control arm says that about 1,490 would have voted without it, so the credit overstates the effect by about five times on the point estimates. The range matters as much as the middle: the interval for the lift excludes zero by a wide margin, and the 95% range for the count is 290 to 474.
How we checked the test
The 10,000 rows are individual voters, but the mailings went to households, and the rows come from 9,536 households with at most four voters each. Voters in one household are not independent of each other, so the plain interval could be too narrow. We recomputed the 95% range with the household as the unit: 5.76 to 9.52 points, against 5.79 to 9.48 with voters as rows. They almost coincide, so the clustering does not change the reading.
That is the check to run on your own file. Analyze at the unit you randomized. If you randomized households and have people as rows, cluster by household, and say which interval you report.
Sizing the control arm
The tool reports the group size needed to detect the observed lift at 80% power and a 5% significance level, using R's power.prop.test on the control rate and the treated rate. We give no size figure for this experiment, because the experiment was large; the formula is what you need before you collect:
- The size per arm grows as the lift you want to detect shrinks, roughly with the inverse of its square.
- It also grows as the rate gets rarer. A rate near 1 percent needs thousands per arm; a rate near 30 percent, as here, needs far fewer for the same relative lift.
- Size from the lift you hope to see, not from the lift you observed. Sizing on an observed lift tells you whether the test you ran could have found it, which is useful after the fact and no help before.
If the size you need is out of reach, size on a more frequent outcome first, or accept a larger smallest-detectable lift.
What this means for your analysis
- Credit and cause are different numbers. Report both, with the same interval logic, and say which one a decision should use.
- The interval is part of the result. A range of 290 to 474 and a range of 0 to 800 call for different decisions at the same middle.
- Randomize before, and fix the window before. A control arm picked after the fact, or an outcome window chosen after you saw the data, is not a control arm.
- Check balance on your own file. Randomization makes the arms alike in expectation; compare the arms on anything you measured before the treatment to catch a broken split.
- Analyze at the unit you randomized. When rows are not the units that were assigned, cluster, and report the clustered range beside the plain one.
- One test answers one question. This one compares a treatment with no treatment. It does not compare two treatments, unless you randomized to both.
Limitations
- Interference. The method assumes one unit's assignment does not change another's outcome. Neighbors who talk to each other, shared households and shared budgets break this. If the treatment also lifts the control arm, the lift is understated.
- Assignment is not exposure. The estimate is intent to treat. If only some of the treated arm read the mailing, the effect per person reached is larger than the effect per person assigned, and these data do not identify it without extra assumptions.
- The window. The result describes the one election measured. Effects that build slowly look smaller; effects that fade look larger in a short window. Fix the window before the treatment.
- Novelty and repetition. A first mailing to a population may behave differently from the tenth.
- Randomization must be real. A control chosen by hand, or by a rule related to the outcome, is not a control. When you cannot randomize, methods such as causal impact or synthetic control estimate the same quantity with more assumptions and wider uncertainty; they are different tools.
- One experiment, one population. The interval describes sampling uncertainty in this experiment, not whether the result carries to a different place, year or message. Voters in Michigan in 2006 are not a general sample of anyone.
How to do it on your own data
- Have one row per unit, a group column naming the arm each unit was randomized to, and a 0 or 1 outcome column. Name the control arm "Control" or "holdout"; otherwise the tool compares against your largest group.
- Compute the rate in each arm and the lift, treated minus control.
- Put a Newcombe interval on the lift, and multiply the lift and its interval by the treated arm's size to get cases caused.
- Test each treated arm against the control, and adjust with Holm if there are several.
- Check the unit: cluster by whatever you randomized if it is not the row.
Every report from the free test names its method and carries the R code that ran it, so each number can be reproduced from the same file. Up to four groups are compared.
Run it on your own file
Upload a file with a group column and a 0 or 1 outcome column and see the lift, its range, the count caused and the checks. Free on up to 10,000 rows. Larger files are randomly sampled, and the report says so. The checklist on this page lists what your file needs.
Run the free incrementality test on your data · The business version of this article
Questions analysts ask
What is the difference between lift and caused?
Lift is the difference in rates between the treated arm and the control, in percentage points. Caused is that difference times the treated arm's size: the number of cases the treatment added. They carry the same interval, scaled. Here the lift is 7.64 points and the count caused is about 382 of 5,000.
Why is a random control arm better than comparing before and after?
Before and after compares the group with its own past, which carries the season, the news and every other change. A random control arm lives through the same window as the treated arm, so those effects cancel in the difference.
Why a Newcombe interval and not the usual normal one?
The normal interval for a difference in proportions can fall below zero or above one for small rates, and under-covers when arms differ in size. The Newcombe interval is built from the Wilson interval of each arm and stays within range. For large arms and moderate rates, as here, the two are close.
Does household clustering matter?
It can. When rows within a household share an outcome, the plain interval is too narrow. Here we checked: the household-clustered range was 5.76 to 9.52 against 5.79 to 9.48, so it did not change the reading. Check it on your own data; do not assume it.
How large should the control arm be?
Large enough to detect the smallest lift worth acting on, at 80% power and 5% significance, using the rate you expect in the control. The smaller the lift and the rarer the outcome, the bigger both arms must be. Fix it before you collect.
Can I use this without a random control?
Not with the same confidence. Without randomization you are estimating the "would have happened anyway" number from something else, and the estimate carries everything else that changed. Methods such as causal impact build a stand-in and say how sure they are.
Is intent to treat the same as the effect of reading the mailing?
No. It is the effect of being assigned to the mailing. Those who read it may respond more, and those who did not may not respond at all. The assigned effect is what a campaign decision rests on, because you choose assignment, not reading.
Three checks on this test
What we checked before we trusted the number.
Like against like
Assignment was random, so the control arm stands in for what the Neighbors group would have done anyway.
A range, not a guess
A 95% range of 5.79 to 9.48 points (Newcombe interval), p below 0.0001. If the test were repeated many times, about 95 percent of ranges built this way would contain the true lift.
Sampling unit
Rows are individual voters from 9,536 households. A household-clustered 95% interval is 5.76 to 9.52 points, so clustering does not change the reading.