SHAP Values: Why Your Model Predicted That Customer Churns

1. Introduction

1.1 The Problem

Aggregate accuracy hides how a model reaches individual predictions. A team that must decide whether to act on a prediction, explain it to a customer or an auditor, or find out why a model fails for a segment needs per-prediction attribution. Global feature importance from tree splits is cheap but says little about any single row.

1.2 What This Paper Covers

• The Shapley value and the additive explanation it gives
• TreeSHAP, KernelSHAP and the assumptions behind them
• A runnable example of local accuracy and of finding a leaky feature
• Limits and good practice

2. Background

2.1 Shapley Values

Shapley (1953) asked how to share a joint payoff fairly among cooperating players. His answer averages each player's marginal contribution over all orders in which players could join, and is the only division that satisfies four axioms: efficiency (the shares add up to the total), symmetry (equal contributors receive equal shares), dummy (a player who adds nothing receives nothing) and additivity.

phi_i = sum over subsets S of the other players of
        [ |S|! * (n - |S| - 1)! / n! ] * [ v(S plus i) - v(S) ]

2.2 From Games to Model Explanations

Lundberg and Lee (2017) treat the features as players and the model's prediction as the payoff: v(S) is the expected model output when only the features in S are known. The result is an additive explanation:

prediction = baseline + SHAP(feature 1) + SHAP(feature 2) + ... + SHAP(feature n)

where the baseline is the average prediction. For a classifier these values are usually in log-odds.

2.3 Computation

Exact Shapley values need the model evaluated on every subset of features, which is infeasible beyond a few features. TreeSHAP (Lundberg and colleagues, 2020) computes them exactly in polynomial time for tree ensembles. KernelSHAP estimates them for any model by weighted regression over sampled subsets and costs far more per explanation. The meaning of "feature absent" matters: some implementations average over a background data set (interventional), others follow the tree's own paths, and the two give different values when features are correlated.

3. Approach

3.1 Reading the Values

For one row, positive values push the prediction up from the baseline and negative values push it down; the sizes show how much. Averaging the absolute values over rows ranks features by how much the model uses them overall. Plots of SHAP value against feature value show direction and shape of use.

3.2 What They Do Not Show

• Not causation. They explain the model's output, whatever the model learned, including spurious patterns.
• Correlated features share credit. Two near-duplicate features split attribution in ways that depend on the model and the background data.
• Not a fairness audit. A protected attribute left out of the model can still influence it through proxy features, and no SHAP value is attributed to the attribute itself.

3.3 Using Them to Audit a Model

A feature that dominates the explanations but could not be known at prediction time, or that records the outcome itself, signals leakage. A feature with a large effect that domain knowledge says should not matter signals a spurious pattern or a data artifact. Both are cheaper to find with SHAP before deployment than after.

4. A Reproducible Example

The code simulates 4,000 customers. Churn depends on days since login, support tickets and tenure; monthly spend is noise. A fifth column, a flag that a cancellation survey was sent, is generated from the outcome itself (a leak). A gradient boosting model is fitted, and SHAP values are computed with TreeSHAP. The code checks that baseline plus the sum of SHAP values equals the model's log-odds for each row, then ranks features by mean absolute SHAP value with and without the leaky column.

import warnings
import numpy as np
import pandas as pd
import shap
from sklearn.ensemble import GradientBoostingClassifier

warnings.filterwarnings("ignore")
rng = np.random.default_rng(31)
n = 4000
df = pd.DataFrame({
    "days_since_login": rng.gamma(2.0, 10.0, n),
    "support_tickets": rng.poisson(1.5, n),
    "tenure_months": rng.integers(1, 60, n),
    "monthly_spend": rng.lognormal(4.0, 0.5, n),
})
# churn depends on the first three columns; monthly_spend is pure noise
logit = -2.0 + 0.05 * df.days_since_login + 0.4 * df.support_tickets - 0.03 * df.tenure_months
churn = rng.random(n) < 1 / (1 + np.exp(-logit))
# a 'leaky' column that records the outcome after the fact (for example a cancellation-survey flag)
df["cancellation_survey_sent"] = np.where(churn, rng.random(n) < 0.9, rng.random(n) < 0.05).astype(int)

model = GradientBoostingClassifier(n_estimators=150, max_depth=3, random_state=0).fit(df, churn)
explainer = shap.TreeExplainer(model)
sv = explainer.shap_values(df)                                   # log-odds contributions, one row per customer

# 1. Local accuracy: baseline + sum of one row's SHAP values equals the model's log-odds for that row
raw = model.decision_function(df)
gap = np.abs(explainer.expected_value + sv.sum(axis=1) - raw).max()
print(f"baseline + sum of SHAP values matches the model's log-odds for every row (gap below 1e-9): {gap < 1e-9}")

# 2. Global view: mean absolute SHAP value per feature
print("mean |SHAP| per feature (log-odds):")
print(pd.Series(np.abs(sv).mean(axis=0), index=df.columns).sort_values(ascending=False).round(3).to_string())

# 3. Without the leaky column the picture changes
clean = df.drop(columns="cancellation_survey_sent")
m2 = GradientBoostingClassifier(n_estimators=150, max_depth=3, random_state=0).fit(clean, churn)
sv2 = shap.TreeExplainer(m2).shap_values(clean)
print("mean |SHAP| per feature without the leaky column:")
print(pd.Series(np.abs(sv2).mean(axis=0), index=clean.columns).sort_values(ascending=False).round(3).to_string())

Output:

baseline + sum of SHAP values matches the model's log-odds for every row (gap below 1e-9): True
mean |SHAP| per feature (log-odds):
cancellation_survey_sent    2.214
days_since_login            0.579
tenure_months               0.324
support_tickets             0.286
monthly_spend               0.184
mean |SHAP| per feature without the leaky column:
days_since_login    0.593
tenure_months       0.410
support_tickets     0.402
monthly_spend       0.124

The additivity check held for every row. The leaky column dominated the explanations, at about four times the average contribution of the next feature. After removing it the model relied on the real drivers, days since login first. Monthly spend, which is pure noise here, still received a small nonzero value, because a flexible model can fit noise and SHAP faithfully reports what the model used. A simulated example like this one only illustrates the mechanism; on real data, treat a surprising top feature as a prompt to investigate how the column is produced.

5. Practical Guidance

6. Conclusion

SHAP values give an additive, per-prediction account of what a model did. They are most useful as an audit tool: in the example, they made a leaked column impossible to miss. They explain the model and not the world, so pair them with domain knowledge and, where action follows, with experiments.