IBM HR Employee Attrition Drivers: When Observational Data Beats Random Assignment

Here's the problem every HR leader faces: you know people are quitting, but you don't know why—at least not with the rigor you'd bring to a pricing experiment or a marketing A/B test. You can't randomize employees to "high overtime" and "low overtime" conditions just to measure causal effects on attrition. Ethics and employment law prevent the clean experimental designs we prefer. Yet decisions still need to be made: Should you cap overtime? Raise salaries across the board or target specific roles? Invest in manager training or fix work-life balance first?

The IBM HR Employee Attrition dataset offers a controlled lab for this messy reality. It contains 1,470 employees tracked across 35 attributes—job role, monthly income, years at company, overtime status, business travel frequency, satisfaction scores, and ultimately whether they left (Attrition = Yes) or stayed (Attrition = No). The baseline attrition rate is 16.1% (237 out of 1,470). This isn't experimental data. It's observational: you're measuring what already happened, not what you made happen. But with the right statistical tools—logistic regression for directional effects and XGBoost with SHAP values for non-linear patterns—you can isolate the independent contribution of each driver and answer the causal question that matters: which levers move attrition, and by how much?

This article walks through an actual attrition drivers analysis of the IBM dataset, card by card. You'll see how to interpret odds ratios from logistic regression, compare them to SHAP feature importance rankings, and identify interaction effects (like "overtime doubles attrition risk in Sales but not in R&D"). Before we draw conclusions, let's check the methodology: we're using two complementary models—logistic regression for transparency and interpretability, XGBoost for capturing complex interactions—and we're controlling for confounders statistically since we can't control them experimentally.

The Research Question: Which Retention Levers Actually Work?

HR teams drown in correlations: "Employees with low satisfaction quit more." "Overtime workers have higher attrition." "Sales reps leave faster than lab technicians." None of that tells you what to do. The research question for attrition driver analysis is causal: If you intervene on variable X (reduce overtime, raise pay, improve manager quality), what's the expected change in attrition rate?

To answer that, you need to separate signal from noise and correlation from causation. Logistic regression does this by estimating the independent effect of each predictor while holding others constant. The output is an odds ratio: "Employees who work overtime have 3.89× the odds of attrition compared to employees who don't, controlling for income, role, tenure, and satisfaction." That's actionable. You can model scenarios: "If we eliminate mandatory overtime for Sales reps, we'd expect attrition to drop from 31% to ~12%, saving $X in recruitment costs."

XGBoost with SHAP values complements this by capturing non-linear effects and interactions. Maybe overtime only matters when combined with frequent business travel, or income has diminishing returns above $10,000/month. SHAP values rank features by their average contribution to predictions, surfacing patterns logistic regression might miss.

Attrition Rate by Segment

Before you fit any model, look at the raw attrition rates stratified by key segments. This table slices the 1,470 employees by Department, Job Level, Business Travel frequency, and Overtime status. The overall attrition rate is 16.1%, but that average masks wide variation. Employees who work overtime show 30.5% attrition vs. 10.4% for those who don't—a nearly 3× difference. Frequent business travelers (those who travel frequently) have 24.9% attrition compared to 14.9% for those who travel rarely and just 8.0% for non-travelers. Job Level 1 (entry-level) employees quit at 26.4%, more than double the 10.5% rate for Level 5 (senior) employees.

These univariate comparisons are your baseline hypotheses. Overtime, business travel, and low job level all correlate with higher attrition. But correlation isn't causation: maybe overtime workers are concentrated in Sales, and Sales roles have inherently higher turnover due to commission structures or job stress. To isolate the independent effect of overtime, you need to control for department, role, and income simultaneously—that's what logistic regression does.

Also notice what doesn't vary much: attrition by department ranges from 13.8% (R&D) to 20.6% (Sales), a 1.5× spread. That's smaller than the 3× spread for overtime. If you had to pick one lever to pull, this table suggests overtime is a stronger candidate than department reassignments. But check the multivariate model before committing budget.

Attrition Rate by Job Role

Job role granularity reveals which positions bleed talent. Sales Representatives top the chart at 39.8% attrition—nearly 4 in 10 quit. Laboratory Technicians follow at 23.9%, then Human Resources at 23.0%. At the other extreme, Research Directors (2.5%) and Managers (4.9%) have negligible attrition. This isn't surprising: senior roles pay more, offer more autonomy, and represent larger sunk investments in firm-specific skills. Junior roles are exit ramps.

Here's the key insight: attrition varies more by role than by any demographic factor (age, gender, education). Sales Rep attrition is 16× higher than Research Director attrition. That magnitude justifies role-specific retention programs—golden handcuffs for Sales reps (e.g., vesting bonuses), career ladders for Lab Techs (clear promotion timelines), better manager training in HR. A one-size-fits-all retention strategy (e.g., "improve company culture") dilutes impact; targeted interventions concentrate resources where attrition risk is highest.

But again, this is univariate. Sales Reps might quit because they work overtime, travel frequently, and earn lower salaries than their effort warrants. Role-specific attrition could be a proxy for working conditions, not an intrinsic feature of the job title. The logistic regression will untangle this by estimating the effect of "being a Sales Rep" after controlling for overtime, income, and travel.

Logistic Regression: Odds Ratios

This chart ranks predictors by their exponentiated coefficient (odds ratio) from the logistic regression model. An odds ratio of 1.0 means no effect; values above 1.0 increase attrition odds, values below 1.0 decrease them. Overtime has an odds ratio of 3.89—employees who work overtime have nearly 4× the odds of quitting, holding everything else constant. That's the largest single effect in the model. BusinessTravel_Travel_Frequently comes in at 2.47×, meaning frequent travelers have 2.5× the odds of attrition compared to non-travelers (after controlling for role, income, overtime, etc.).

On the protective side, MonthlyIncome has an odds ratio of 0.9996 per dollar—meaning each additional $1,000 in monthly income multiplies attrition odds by 0.9996^1000 ≈ 0.67, or a 33% reduction. YearsAtCompany (odds ratio ~0.95 per year) shows that every additional year of tenure reduces attrition odds by ~5%. These are independent effects: the model estimates the income effect after adjusting for job role, level, and overtime. So you're not just seeing "senior people earn more and quit less"—you're isolating the marginal return to paying someone more within their role.

What's your sample size? Is this test adequately powered? With 237 attrition events and 15 predictors, you have ~16 events per variable—enough for stable estimates. The confidence intervals (not shown on this chart but visible in the full coefficient table) will tell you which effects are statistically significant. If the 95% CI for an odds ratio crosses 1.0, you can't rule out zero effect; treat that predictor as uncertain.

Here's how to use these odds ratios: imagine you're designing a retention experiment (or a policy intervention that mimics one). If you eliminate mandatory overtime for high-risk employees, you'd expect their attrition odds to drop by a factor of 3.89, translating to a percentage-point reduction you can forecast using the baseline rate. That's the actionable output HR needs: not "overtime correlates with quitting," but "removing overtime reduces quit risk by X%, generating $Y in retained-employee value."

Logistic Regression: Full Coefficient Table

This is the table you show to the CFO or legal counsel when they ask, "How confident are we that overtime actually causes higher attrition?" Each row is a predictor; columns show the coefficient, standard error, z-statistic, p-value, and 95% confidence interval. OverTime_Yes has a coefficient of +1.36 (log-odds scale), std error 0.18, z = 7.5, p < 0.001, with a confidence interval that doesn't come close to zero. That's overwhelming evidence for a large effect. Same for BusinessTravel_Travel_Frequently (coef +0.90, p < 0.001) and MonthlyIncome (coef –0.0004, p < 0.001).

Now look at YearsAtCompany: coefficient –0.05, p < 0.001. For every additional year at the company, log-odds of attrition drop by 0.05, or odds multiply by e^(–0.05) ≈ 0.95. Over 10 years, that's 0.95^10 ≈ 0.60—a 40% reduction in attrition odds for a 10-year veteran vs. a new hire, after controlling for income and role. This quantifies the "golden handcuffs" effect of tenure and firm-specific capital.

Some predictors are not significant. If you see p-values above 0.05 (not shown here but watch for them in your own data), those variables didn't add explanatory power after accounting for the others. Drop them from your model or at least don't build retention programs around them. The goal isn't to include every possible predictor—it's to identify the levers you can pull (overtime, travel policy, compensation) and estimate their effect sizes with statistical rigor.

This table is your evidence base for causal claims. You can't say "we proved overtime causes attrition" with the certainty of a randomized experiment, but you can say "after controlling for 14 other factors—role, income, tenure, satisfaction, travel—overtime independently predicts a 3.89× increase in attrition odds, p < 0.001, with a confidence interval of [2.7, 5.6]." That's strong enough to justify a pilot program capping overtime hours and measuring the downstream attrition change.

XGBoost SHAP Feature Importance

SHAP (SHapley Additive exPlanations) values measure each feature's contribution to XGBoost predictions, averaged across all employees. Unlike logistic regression, XGBoost captures non-linear relationships and interactions—threshold effects, diminishing returns, conditional dependencies. The chart ranks features by mean absolute SHAP value, which reflects "how much does this feature move predictions, on average?"

MonthlyIncome tops the list with a mean |SHAP| around 0.20, followed by OverTime (~0.18), Age (~0.15), and YearsAtCompany (~0.13). This ranking differs slightly from logistic regression: logistic regression ranked OverTime #1 (odds ratio 3.89), while SHAP ranks MonthlyIncome #1. Why the discrepancy? Because XGBoost detected non-linearities in income: attrition drops sharply as income rises from $2,000 to $5,000/month, then plateaus. Logistic regression assumes a constant log-odds effect per dollar; XGBoost learns the threshold. For employees below $4,000/month, income might be the dominant driver; for those above $8,000, income matters less than overtime or work-life balance.

JobRole and Department also rank higher in SHAP than in logistic regression. That suggests interaction effects: maybe being a Sales Rep increases attrition more when combined with overtime than when combined with low travel. Logistic regression estimates an average "Sales Rep effect"; SHAP captures how that effect varies depending on other features.

Here's the practical takeaway: use logistic regression for your executive summary and policy memos (clear, interpretable odds ratios), then use SHAP to refine targeting. If SHAP shows MonthlyIncome has the highest importance, dig into the distribution: at what income threshold does attrition risk stabilize? That threshold becomes your compensation floor for retention-sensitive roles. If SHAP elevates Age or YearsAtCompany, consider age- or tenure-specific retention bonuses rather than uniform raises.

Overtime × Department Attrition Heatmap

This heatmap cross-tabulates Overtime (Yes/No) by Department (HR, R&D, Sales) and fills cells with attrition rate. It's a simple 2×3 grid that answers a critical question: Is the overtime effect uniform across departments, or is it concentrated in one area? If overtime-driven attrition is a Sales-only problem, you'd target overtime reduction there; if it's company-wide, you'd need a universal policy change.

The data shows: In Sales, overtime employees have 38.5% attrition vs. 16.7% for non-overtime employees—a 2.3× difference. In R&D, overtime employees show 22.2% attrition vs. 9.5% for non-overtime—a 2.3× ratio again. In HR, the pattern holds: 25.0% vs. 10.5%, roughly 2.4×. So the overtime effect is multiplicative and consistent across departments. It's not unique to Sales; it's a universal stressor. That means an overtime reduction policy would yield attrition savings in all three departments, not just one.

But also notice the baseline differences: Sales non-overtime attrition (16.7%) is higher than R&D non-overtime attrition (9.5%). Sales has an inherently higher quit rate even without overtime—possibly due to commission volatility, customer-facing stress, or external recruiting pressure. So Sales needs both overtime reduction and role-specific interventions (e.g., better commission structures, manager training). R&D's lower baseline suggests cultural or job-design factors that reduce turnover; maybe R&D employees value intellectual challenge and have fewer outside options. Use those insights to differentiate retention strategies by department while addressing overtime as a cross-cutting lever.

Interaction heatmaps like this are your first defense against Simpson's Paradox and confounding. If you found that overtime increased attrition only in Sales, you'd suspect confounding: maybe Sales hires younger employees who were going to quit anyway, and overtime is correlated but not causal. The fact that the effect replicates across departments strengthens the causal interpretation.

How to Interpret Your Results

You've seen six analysis cards: univariate attrition rates by segment and role, logistic regression odds ratios and full coefficients, XGBoost SHAP importance, and an overtime × department heatmap. Now translate those numbers into decisions. Here's the framework:

1. Prioritize High-Magnitude, High-Confidence Levers

Overtime has an odds ratio of 3.89, p < 0.001, and replicates across departments. That's high magnitude (4× effect) and high confidence (tight CI, low p-value). It's your top retention lever. Monthly income has a smaller per-dollar effect but applies to everyone; increasing pay by $1,000/month for at-risk employees could cut attrition odds by ~33%. YearsAtCompany has a protective effect, but you can't force tenure—you can only create conditions (pay, culture, career growth) that make people want to stay long enough to accumulate it.

2. Distinguish Between Levers You Can Pull and Indicators You Monitor

You can change overtime policy, adjust compensation, and reduce business travel. You can't directly change Age or YearsAtCompany—those are outcomes of retention, not inputs. If Age or YearsAtCompany rank high in your SHAP plot, that tells you tenure and experience matter, which implies you should focus on early-career retention (because if people quit in Year 1, they never reach the protective tenure threshold). But don't write "Age" into your action plan; write "retention bonuses vesting at 2 years" or "structured career ladders for junior roles."

3. Check for Interaction Effects Before Scaling Interventions

The heatmap showed overtime effects are consistent across departments, so an overtime cap is a universal lever. But if SHAP ranks Department or JobRole high, investigate whether certain roles need tailored interventions. Sales Reps have 39.8% attrition even before considering overtime—maybe they need better manager support, clearer quota-setting, or commission smoothing to reduce volatility. Lab Technicians at 23.9% might benefit from formalized promotion tracks so they see a future beyond "technician." Use SHAP and interaction heatmaps to identify these subgroups, then design A/B tests (or quasi-experiments with matched controls) to validate targeted interventions.

4. Build Business Cases with Counterfactual Scenarios

CFOs don't care about p-values; they care about ROI. Translate odds ratios into dollar terms. If eliminating mandatory overtime reduces Sales Rep attrition from 39.8% to (39.8% ÷ 3.89) ≈ 10.2%, and you have 100 Sales Reps, you'd retain an extra 30 employees per year. If average recruitment + training cost is $15,000 per hire, that's $450,000 saved annually. Compare that to the cost of hiring additional headcount to cover reduced overtime hours. If the payroll increase is $200,000/year, you net $250,000—a clear positive ROI. Run that calculation for each proposed intervention using the logistic regression coefficients as your effect size estimates.

5. Plan a Validation Study (Even If It's Not a True Experiment)

Observational analysis identifies candidates for intervention; it doesn't prove causation with experimental certainty. Once you've chosen a lever (e.g., overtime reduction), implement it in a pilot group and track attrition over 6–12 months. Ideally, use a matched control group (employees similar in role, tenure, income, but not subject to the policy change) or a staggered rollout (different departments adopt the policy at different times) to approximate experimental conditions. Measure the before-after difference and compare it to your logistic regression prediction. If observed attrition drops by 50% and your model predicted 60%, you've validated the model and can scale confidently. If the drop is only 10%, revisit your assumptions—maybe unobserved confounders (manager quality, team culture) were driving the original association.

When Logistic Regression and SHAP Disagree, Investigate Non-Linearities

Logistic regression ranked OverTime #1 (odds ratio 3.89); SHAP ranked MonthlyIncome #1 (mean |SHAP| 0.20). That's not a contradiction—it's a signal to dig deeper. SHAP captures the total contribution of MonthlyIncome across its full range, including threshold effects and diminishing returns. Logistic regression gives you a single coefficient assuming a constant log-linear effect per dollar. If income has a step-function shape (big attrition drop from $2K to $5K, then flat above $8K), SHAP will rank it high because the big drop dominates the average, while logistic regression will estimate a modest linear slope that averages over the whole range.

To reconcile, plot attrition rate by income bins (e.g., $0–$3K, $3K–$6K, $6K–$10K, $10K+). If you see a hockey-stick curve, you've found a non-linearity. Policy implication: raising salaries from $2K to $5K yields massive retention gains; raising from $10K to $13K yields little. Target pay raises at the low end; use other levers (career development, autonomy, recognition) at the high end where marginal income has diminishing returns.

Similarly, if SHAP elevates Department or JobRole above their logistic regression ranking, check for interaction effects. Maybe JobRole_SalesRep has a large SHAP value because its effect depends on overtime: Sales Reps who work overtime quit at 50%, but Sales Reps without overtime quit at 15%. Logistic regression estimates separate main effects for JobRole and OverTime; SHAP captures the amplified interaction. Use interaction heatmaps (like Overtime × Department) to surface these patterns, then add interaction terms to your logistic regression (JobRole_SalesRep × OverTime) to quantify the effect and test significance.

The Statistical Power Question: Could This Be Noise?

Before you commit six figures to an overtime reduction program, ask: What's the minimum effect size we could reliably detect with this sample? Power analysis answers that. With 237 attrition events, 1,233 non-attrition observations, and a logistic regression with 15 predictors, you have 80% power to detect an odds ratio around 1.5–1.8 (depending on baseline rates and predictor correlations). The observed overtime odds ratio of 3.89 is well above that threshold—you're overpowered, meaning the effect is large enough that even a smaller sample would catch it.

But if you were analyzing a rare outcome (e.g., "attrition among senior executives," where you have only 20 events), your minimum detectable effect size would balloon to odds ratios of 3–5. Anything smaller would be statistically indistinguishable from noise. In that scenario, you'd need to either (a) collect more data (wait another year, pool data across divisions), (b) simplify your model (use only 2–3 predictors instead of 15), or (c) accept that you can only detect very large effects and might miss moderate ones.

For the IBM dataset, power isn't a limiting factor—237 events is plenty. But when you run this on your own HR data, check your attrition event count first. If N_events < 100, you're in the danger zone for overfitting and unstable estimates. Focus on univariate comparisons and simple models; resist the temptation to throw 20 variables into XGBoost and cherry-pick whatever looks significant.

What to Do When You Can't Randomize

Employment law, ethics, and practicality prevent you from randomly assigning employees to "overtime" vs. "no overtime" just to measure causal effects. You're stuck with observational data. Here's how to extract causal inference anyway:

1. Propensity Score Matching. Instead of controlling for confounders in a regression, match each overtime employee to a similar non-overtime employee (same role, tenure, income, satisfaction) and compare their attrition rates. This mimics randomization by creating balanced groups. If matched overtime employees still have 3× higher attrition, you've strengthened the causal claim.

2. Difference-in-Differences. If overtime policy changes over time or varies by department, use a before-after comparison with a control group. Example: Sales adopts a no-mandatory-overtime policy in Q3 2025, while R&D continues as usual. Compare Sales attrition trend (before vs. after Q3) to R&D attrition trend (same period). If Sales shows a relative drop, that's causal evidence.

3. Instrumental Variables. Find a variable that affects overtime but doesn't directly affect attrition (an "instrument"). Example: distance from home to office might predict overtime (longer commutes = more willingness to stay late) but not directly drive attrition (unless through overtime). Use the instrument to isolate exogenous variation in overtime. This is advanced and requires strong assumptions, but it's the closest you get to experimental causal inference with observational data.

None of these methods are as clean as a randomized experiment, but they're vastly better than ignoring confounders. Apply them when the stakes are high and you need defensible causal claims for board-level decisions or legal compliance.

Common Pitfalls and How to Avoid Them

Pitfall 1: Treating Prediction Accuracy as Evidence of Causal Validity

XGBoost might achieve 85% AUC predicting who quits, but high AUC doesn't mean the model identified causal drivers. Maybe it's picking up on proxies: "employees who update their LinkedIn profiles quit more"—true, but changing LinkedIn isn't a lever you can pull. Focus on interpretable models (logistic regression) and validate causal claims with domain knowledge and quasi-experimental designs.

Pitfall 2: Ignoring Base Rates When Interpreting Odds Ratios

An odds ratio of 3.89 sounds huge, but it translates differently depending on baseline attrition. If baseline is 1%, a 4× increase yields 4% attrition—still low. If baseline is 40%, a 4× increase would imply >100% attrition (impossible), signaling model misspecification. Always convert odds ratios back to probabilities using the baseline rate: P(attrition | overtime) = baseline × OR / (1 + baseline × (OR – 1)). For 16.1% baseline and OR = 3.89, that's ~44% attrition with overtime vs. ~16% without.

Pitfall 3: Over-Controlling and Washing Out Real Effects

If you include too many correlated predictors, you risk "controlling away" the effect you care about. Example: if you add "employee engagement score" (which is probably caused by overtime), you'll underestimate the overtime coefficient because engagement mediates the effect. Solution: draw a causal diagram (DAG) to distinguish confounders (variables that cause both overtime and attrition) from mediators (variables caused by overtime that then cause attrition). Control for confounders; don't control for mediators if you want the total effect of overtime.

Pitfall 4: Using Attrition Analysis to Justify Layoffs

Attrition driver analysis identifies retention levers—variables that predict voluntary quits. Do not use it to predict "who should we fire?" That's a different problem (performance prediction) with different ethics and legal constraints. If you build a "flight risk score" to preemptively terminate high-attrition employees, you'll face wrongful termination lawsuits and destroy trust. Use these models to retain talent, not to preempt it.

Frequently Asked Questions

Why can't you run a randomized experiment on employee attrition?

You can't ethically or practically randomize employees to "high overtime" vs. "low overtime" conditions just to measure attrition. HR analytics relies on observational data—you measure what actually happened, control for confounders statistically, and look for actionable patterns. Logistic regression and SHAP values let you isolate the independent effect of each driver after accounting for correlations.

What's the difference between logistic regression odds ratios and SHAP values?

Logistic regression gives you directional, interpretable coefficients: "Overtime increases attrition odds by 3.89×, holding everything else constant." It assumes linear relationships. SHAP values from XGBoost capture non-linear patterns and interactions: "Overtime matters more in Sales than R&D." Use logistic regression for clear talking points; use SHAP when you suspect thresholds or interaction effects.

How large a sample do you need for employee attrition analysis?

The IBM dataset has 1,470 employees with 16.1% attrition (237 events). That's enough to fit a logistic regression with ~10-15 predictors and detect moderate effect sizes. Rule of thumb: you need at least 10–15 attrition events per predictor variable. If you have fewer than 100 attrition events total, focus on univariate comparisons and resist the urge to build complex models.

Should you treat attrition analysis as a prediction problem or a causal inference problem?

Both—but prioritize causal inference. Predictive accuracy (AUC, precision/recall) tells you if you can forecast who quits. Causal inference (logistic coefficients, propensity matching) tells you which levers to pull. Most HR teams need the latter: "If we reduce mandatory overtime in Sales by 50%, how much does attrition drop?" Answer that question first, then build risk scores for targeting interventions.

What do you do when SHAP and logistic regression disagree on which variables matter?

Investigate interaction effects and non-linearities. If SHAP ranks "MonthlyIncome" higher than logistic regression does, check for threshold effects (e.g., attrition spikes below $3,000/month but plateaus above $8,000). If SHAP ranks "Department" higher, look for Department × Overtime interactions. Use logistic regression for your executive summary; use SHAP to find the nuances that justify targeted programs.

Final Takeaway: From Correlation to Action

The IBM HR Employee Attrition dataset demonstrates what rigorous observational analysis looks like when experiments are off the table. You can't randomize overtime, but you can control for confounders statistically, estimate independent effects with logistic regression, validate non-linearities with XGBoost SHAP, and check for interactions with segment-level heatmaps. The result isn't proof of causation—that requires an experiment—but it's actionable evidence strong enough to justify pilot programs, budget allocation, and policy changes.

Overtime emerges as the single largest lever: 3.89× odds ratio, consistent across departments, statistically significant with tight confidence intervals. If you're an HR leader looking for one intervention to move the needle on attrition, that's it. Monthly income shows threshold effects—big returns at the low end, diminishing returns above $8K. Job role and department matter, but mostly as indicators of where to target interventions (Sales needs overtime + commission fixes; Lab Techs need career ladders), not as levers themselves.

Upload your own employee data—attrition status, job role, salary, tenure, overtime, satisfaction scores—and run this analysis in 60 seconds. You'll get logistic regression odds ratios, SHAP feature importance, segment attrition rates, and interaction heatmaps. No coding, no statistical software, no PhD required. Just clean data and a question: which retention levers actually work?