Cohort Analysis for Retention and Churn: A Method Guide

1. Introduction

1.1 Why Cohorts

A blended retention figure mixes old and new customers and hides trends. Grouping users by start month (or week) and following each group by age shows whether newer customers retain better or worse than older ones, and which acquisition channels or product changes line up with the change.

1.2 What This Paper Covers

• How to define retention and churn so they can be compared
• How monthly churn compounds over time
• How to test whether two cohorts really differ, with runnable examples
• How cohort size affects false alarms
• Guidance for reading cohort tables and curves

2. Background

2.1 The Cohort Table

A cohort table has one row per start period and one column per age (week 1, week 2, and so on). Each cell is the share of that cohort still active at that age. Reading across a row shows one cohort's decay; reading down a column compares cohorts at the same age, which is the fair comparison. Cells for ages a cohort has not reached yet are blank, and should stay blank.

2.2 Retention and Churn

Retention at age k is the share of the starting group still active; churn in a period is the share of those active at the start of the period who leave during it. State the activity definition (a purchase, a login, a paid subscription), because retention numbers from different definitions cannot be compared.

2.3 Curve Shapes

Some cohort curves keep falling at a steady rate; others fall early and then level off at a stable share. The two can show the same retention at day 30 and very different retention at day 90, so one early number is a weak guide to the long run. Plotting the full curve and its uncertainty is more informative than quoting one point.

3. Approach

3.1 Intervals for a Retention Rate

A retention rate from n users has sampling error. For a proportion p, the standard error is the square root of p(1 - p) / n, and a rough 95% interval is p plus or minus 1.96 times that. The interval narrows with the square root of n, so cohorts of a few hundred users give intervals several points wide.

3.2 Comparing Two Cohorts

For two independent cohorts, the standard error of the difference combines both. A two-proportion z-test asks whether the observed gap would be unusual if the true rates were equal. The p-value is the probability of data at least this extreme if there were no real difference; a small p-value is evidence against no difference, not proof of an effect, and a large one is not proof of no effect. Cohorts that overlap in membership, or that were exposed to different campaigns, need more care than this simple test gives.

3.3 Churn Compounds

With a constant monthly churn c, the share still active after n months is (1 - c)ⁿ, and the average lifetime is 1 / c months. Real churn is rarely constant, because early months usually lose more than later ones, so treat these as illustrations of the arithmetic.

3.4 Incomplete Observation

Customers who started recently have not had time to churn. Compare cohorts only at ages that all of them have reached, or use a survival method such as Kaplan-Meier that handles customers still active at the end of observation.

4. Worked Calculations

4.1 Is a 5-Point Gap Real?

Two cohorts of 850 users, with 58% and 63% active in week 2:

import numpy as np
from scipy import stats

# Two weekly cohorts of 850 users each: 58% and 63% still active in week 2
n1 = n2 = 850
p1, p2 = 0.58, 0.63
diff = p2 - p1
se = np.sqrt(p1 * (1 - p1) / n1 + p2 * (1 - p2) / n2)
lo, hi = diff - 1.96 * se, diff + 1.96 * se
pooled = (p1 * n1 + p2 * n2) / (n1 + n2)
z = diff / np.sqrt(pooled * (1 - pooled) * (1 / n1 + 1 / n2))
print(f"difference {diff*100:.1f} points, standard error {se*100:.2f} points")
print(f"95% interval for the difference: {lo*100:.2f} to {hi*100:.2f} points")
print(f"two-sided p-value: {2 * stats.norm.sf(abs(z)):.3f}")

difference 5.0 points, standard error 2.37 points
95% interval for the difference: 0.36 to 9.64 points
two-sided p-value: 0.035

The gap is statistically distinguishable from zero at the usual 5% level, but the interval runs from under half a point to nearly ten points, so how large it is remains unclear. Whether it justifies an intervention depends on what the intervention costs and what the gap is worth.

4.2 How Often Does Chance Produce a Gap?

Here the true retention rate is the same, 60%, for both cohorts in every pair, and the code computes exactly, from the binomial distribution, how often the observed rates still differ by 5 points or more.

import numpy as np
from scipy import stats

# How big a gap between two cohorts appears by chance when the true retention rate is the same?
# Exact calculation: both cohorts have the same true rate, so each retained count is Binomial(n, 0.60).
true_rate = 0.60
print("cohort size   share of pairs whose retention differs by 5 points or more")
for n in (100, 300, 850, 3000):
    pmf = stats.binom.pmf(np.arange(n + 1), n, true_rate)
    counts = np.arange(n + 1)
    gap_counts = np.abs(counts[:, None] - counts[None, :])
    big = gap_counts >= 0.05 * n - 1e-9               # tolerance so a gap of exactly 5 points counts
    print(f"{n:11d}   {np.sum(pmf[:, None] * pmf[None, :] * big):.1%}")

cohort size   share of pairs whose retention differs by 5 points or more
        100   51.6%
        300   22.7%
        850   3.5%
       3000   0.0%

Small cohorts produce large gaps by chance alone, which is why a tidy table of week-2 numbers for cohorts of a hundred users should not trigger action by itself.

4.3 How Churn Compounds

# Monthly churn compounds: the share of a starting group still active after n months is (1 - churn) ** n
for churn in (0.05, 0.07):
    after_14 = (1 - churn) ** 14
    after_20 = (1 - churn) ** 20
    print(f"monthly churn {churn:.0%}: {after_14:.0%} still active after 14 months, "
          f"{after_20:.0%} after 20 months, average lifetime 1/churn = {1 / churn:.1f} months")

monthly churn 5%: 49% still active after 14 months, 36% after 20 months, average lifetime 1/churn = 20.0 months
monthly churn 7%: 36% still active after 14 months, 23% after 20 months, average lifetime 1/churn = 14.3 months

The reciprocal of monthly churn is the average lifetime of a customer under constant churn. It is not the time until the whole starting group has gone: a sizeable share remains after that many months.

5. Practical Guidance

6. Conclusion

Cohort tables are useful when read at the same age, with intervals and with attention to cohort size. The worked calculations show that a 5-point gap can be real or can be chance depending on how many users each cohort holds, and that churn arithmetic compounds in a way the reciprocal rule hides. Act on gaps that the numbers support, and collect more data before acting on the rest.