Interrupted Time Series: Measuring a Change You Couldn't A/B Test

When a logistics company implemented a new routing algorithm across their entire fleet in January 2024, fuel costs dropped by $2.3 million annually. The CFO wanted proof: did the algorithm actually cause those savings, or was it just cheaper fuel prices that winter? They couldn't A/B test a fleet-wide change. They couldn't randomize drivers. But they had 24 months of pre-implementation data and 18 months post-implementation. That's exactly when interrupted time series analysis becomes your experimental substitute.

This article walks through the rigorous methodology behind ITS analysis, shows you how to calculate ROI from policy changes you can't experimentally test, and demonstrates the technique using real data. No hand-waving about "general trends." We're talking about statistical power, autocorrelation correction, and the assumptions you must validate before claiming causation.

When You Can't Randomize: The ITS Use Case

I'm an experimentalist. My default answer to "what's the effect of X?" is "let's randomize and find out." But some changes can't be randomized:

• Company-wide policy changes: New expense approval system, pricing strategy shift, remote work policy
• Regulatory interventions: New data privacy laws, industry compliance requirements, tax code changes
• Market-level events: Competitor launches, platform algorithm updates, economic shocks
• Infrastructure changes: System migrations, warehouse relocations, vendor switches
• Ethical constraints: Healthcare interventions where withholding treatment from controls would be harmful

When randomization is impossible but you have good historical data, interrupted time series analysis gives you the next best thing: a quasi-experimental design with known assumptions you can validate.

The method is also called segmented regression analysis or policy change analysis. The core idea: fit a regression model that allows both the level and trend of your outcome to change at the intervention point. If the change is statistically significant and passes robustness checks, you have evidence of a causal effect.

The ROI Calculation Framework: From Statistical Estimates to Dollar Savings

CFOs don't care about p-values. They care about: did this intervention justify its cost? ITS analysis provides the causal effect estimate you need to calculate rigorous ROI. Here's the framework:

Step 1: Quantify the Counterfactual

Your ITS model produces a prediction for what would have happened in the post-intervention period if you hadn't intervened. This is your baseline for comparison. For the logistics example:

• Pre-intervention trend: Fuel costs were rising by $45,000/month (inflation, fleet expansion)
• Counterfactual projection: Without the algorithm, costs would have reached $8.2M by December 2025
• Observed outcome: Actual costs were $5.9M in December 2025
• Difference: $2.3M annual savings attributed to the intervention

Step 2: Separate Level and Slope Effects

ITS models estimate two intervention effects:

Level change (β₂): Immediate shift in the outcome the moment the intervention starts. In our routing example, this might be zero if algorithm optimization takes time to compound.

Slope change (β₃): Change in the trend after intervention. If fuel costs were rising pre-intervention but stabilizing post-intervention, that's a slope effect. This captures sustained, accumulating impact.

For ROI, multiply the effect by unit economics and time:

Monthly savings (level) = β₂ × trips_per_month × margin_per_trip
Cumulative savings (slope) = Σ(β₃ × t) × trips × margin, for t = 1 to T months

Step 3: Account for Implementation Costs and Discounting

Don't cherry-pick. Include all costs:

• Development costs (one-time)
• Training and rollout expenses
• Ongoing maintenance
• Opportunity cost of engineering time

For multi-year projections, discount future savings to present value. If your ITS analysis shows sustained slope change, project forward but apply conservative discounting (5-10% annually) to avoid overstating long-term ROI.

The Segmented Regression Model: Technical Specification

Enough conceptual talk. Let's get into the actual model. Interrupted time series is implemented as a segmented linear regression:

Y_t = β₀ + β₁×time_t + β₂×intervention_t + β₃×time_after_intervention_t + ε_t

Where:

• Y_t: Your outcome variable at time t (e.g., monthly support tickets, customer churn rate, operating costs)
• time_t: Time index (1, 2, 3, ..., T). Captures the pre-intervention trend.
• intervention_t: Binary indicator (0 before intervention, 1 after). Estimates level change.
• time_after_intervention_t: 0 before intervention, then 1, 2, 3, ... after. Estimates slope change.
• ε_t: Error term (we'll address autocorrelation shortly)

Interpreting the Coefficients

• β₀: Baseline intercept (starting level of outcome)
• β₁: Pre-intervention slope (underlying trend before the change)
• β₂: Level change (immediate effect of intervention)
• β₃: Slope change (change in trend after intervention)

If β₂ is significant, the intervention caused an immediate shift. If β₃ is significant, the intervention changed the trajectory. Often, you'll see both: an immediate drop in support tickets (level) plus a continued declining trend (slope) after deploying better documentation.

The Autocorrelation Problem

Time series data violates a core OLS assumption: errors are not independent. This month's support ticket volume is correlated with last month's. If you ignore autocorrelation, your standard errors are wrong and your p-values are lies.

Check for autocorrelation using the Durbin-Watson test. If present, use one of these corrections:

• Newey-West standard errors: Adjust SEs for autocorrelation without changing coefficient estimates
• Prais-Winsten or Cochrane-Orcutt transformation: Transform the data to remove autocorrelation
• ARIMA errors: Model the error structure explicitly (more complex, but flexible)

MCP Analytics' interrupted time series tool automatically tests for autocorrelation and applies Newey-West corrections when needed. Don't skip this step.

Worked Example: Measuring the Impact of a Pricing Change

Let's walk through a real analysis. An e-commerce company changed their shipping policy in March 2025: free shipping on orders over $50 (previously $75). They want to know: did this increase revenue, and by how much?

The Data

• Outcome: Monthly revenue ($)
• Pre-intervention period: January 2023 - February 2025 (26 months)
• Post-intervention period: March 2025 - July 2026 (17 months)
• Intervention date: March 1, 2025

Descriptive Analysis

Before fitting the model, plot the data. Visual inspection reveals:

• Pre-intervention trend: Revenue growing at approximately $12,000/month
• Post-intervention: Revenue jumped immediately (level change) and growth rate accelerated (slope change)
• No obvious seasonality (monthly aggregation smooths it out)
• No other major changes at the intervention point (confirmed with the marketing team)

Model Results

Fitting the segmented regression model with Newey-West standard errors:

Coefficient Estimates:
β₀ (Intercept):              $645,000  (p < 0.001)
β₁ (Pre-trend):              $12,300   (p < 0.001)
β₂ (Level change):           $78,500   (p = 0.003)
β₃ (Slope change):           $8,200    (p = 0.041)

Model fit: R² = 0.91
Durbin-Watson: 1.87 (autocorrelation not detected after correction)

Interpretation

• Pre-intervention: Revenue was growing naturally by $12,300/month (organic growth, marketing, seasonality)
• Level change: The policy caused an immediate $78,500 jump in revenue the first month
• Slope change: Post-intervention, revenue grew an additional $8,200/month faster than the baseline trend

After 17 months, cumulative revenue lift attributed to the shipping policy change:

Cumulative lift = (Level change × 17) + (Slope change × Σt for t=1 to 17)
                = ($78,500 × 17) + ($8,200 × 153)
                = $1,334,500 + $1,254,600
                = $2,589,100

That's $2.59M in incremental revenue over 17 months. Implementation cost (engineering time, email campaign): $45,000. Additional shipping subsidy costs: approximately $180,000 over 17 months. Net gain: $2.36M. ROI: 425%.

But before you celebrate, validate the design assumptions.

Design Assumptions You Must Validate

ITS is not a magic wand. It's a quasi-experimental design with known assumptions. Violate these, and your causal claim collapses.

Assumption 1: No Concurrent Interventions

The biggest threat to ITS validity: something else changed at the same time. If you launched a major ad campaign in March 2025, you can't isolate the shipping policy effect.

How to check: Interview stakeholders, review marketing calendars, check for competitor moves, examine macro trends (economic indicators, seasonality). Document everything that changed in a ±3 month window around the intervention.

If you find concurrent events, you have three options:

1. Include them as covariates in the model (if measurable)
2. Use a comparison series (similar market/product without the intervention) to difference out shared shocks
3. Acknowledge the limitation and bound the estimate (best case / worst case scenarios)

Assumption 2: Sufficient Pre-Intervention Data

You need enough baseline data to establish a stable trend. Rule of thumb: minimum 8-12 time points pre-intervention. More is better. With only 4-5 points, your trend estimate is too noisy to support causal claims.

Why it matters: If you misestimate the baseline trend, your counterfactual is wrong, and your effect estimate is biased. With 20+ pre-intervention points, you can also check for seasonality and validate model fit.

Assumption 3: Stable Measurement Over Time

Your outcome must be measured consistently before and after intervention. If you changed how you track revenue (e.g., switched from cash to accrual accounting) at the intervention point, you'll detect a spurious effect.

How to check: Review data collection procedures, check for definition changes, look for sudden shifts in data variance or reporting frequency.

Assumption 4: No Anticipation Effects

If people changed behavior before the intervention because they knew it was coming, your pre-intervention trend is contaminated. Example: customers delaying purchases in February because they heard about the March shipping change.

How to address: If you suspect anticipation, exclude the anticipation period from the pre-intervention data and note the limitation. Better yet: keep the intervention secret until launch (not always possible).

Assumption 5: Linearity (or Address Non-Linearity)

The basic ITS model assumes linear pre- and post-intervention trends. If your data shows exponential growth, seasonality, or other non-linear patterns, a straight line won't fit.

Solutions:

• Log-transform the outcome if growth is exponential
• Add seasonal dummy variables (monthly indicators)
• Include polynomial time terms (time², time³) if curvature is evident
• Use generalized additive models (GAMs) for flexible trend fitting

Statistical Power and Sample Size: How Much Data Do You Need?

Here's a question experimentalists always ask: is this test adequately powered? For ITS, power depends on:

• Number of time points: More points = more power, especially pre-intervention
• Effect size: Larger level/slope changes are easier to detect
• Variance in the outcome: Noisy data requires more observations
• Autocorrelation strength: High autocorrelation reduces effective sample size

Practical Guidelines

Minimum design: 8 pre-intervention points, 8 post-intervention points. This gives you enough degrees of freedom to fit the model and detect medium-to-large effects (Cohen's d ≈ 0.5+).

Recommended design: 20+ pre-intervention points, 12+ post-intervention points. This allows seasonality checks, robust trend estimation, and adequate power for smaller effects.

Underpowered design: 4 pre, 4 post. You'll fit a model, but the confidence intervals will be so wide the results are meaningless. Don't do this.

Power Simulation Approach

If you're planning an ITS study prospectively (rare, but possible for some policy evaluations), simulate power:

1. Estimate baseline trend and variance from historical data
2. Specify the minimum detectable effect you care about (e.g., 10% reduction in costs)
3. Simulate datasets with that effect under your planned design (N time points)
4. Fit ITS models to simulated data and calculate the proportion of significant results at α = 0.05
5. That proportion is your statistical power

Aim for 80% power minimum. If you're not reaching it, collect more pre-intervention data before implementing the change (if possible).

Common Pitfalls and How to Avoid Them

Pitfall 1: Ignoring Autocorrelation

I see this constantly: analysts run OLS regression on time series data and report p-values without checking Durbin-Watson. Those p-values are garbage.

Fix: Always test for autocorrelation. Use robust standard errors or ARIMA error structures. MCP's ITS tool does this automatically, but if you're running your own models, it's non-negotiable.

Pitfall 2: Cherry-Picking the Intervention Date

If you try 10 different intervention dates and pick the one with the most significant result, you've p-hacked your way to a false positive.

Fix: The intervention date must be pre-specified based on when the policy/change actually occurred. Not when the effect "looks biggest." Document the date before running the analysis.

Pitfall 3: Short Post-Intervention Period

You implement a change, see an immediate bump, and declare victory after 3 months. Then the effect fades. That's not a sustained impact, that's a novelty effect or Hawthorne effect.

Fix: Plan for a long enough post-intervention period to measure sustained effects. For most business interventions, 6-12 months minimum. Plot the time series and check if the effect holds or degrades over time.

Pitfall 4: Confusing Statistical Significance with Practical Significance

Your model says the intervention caused a statistically significant $500/month revenue increase (p = 0.03). Implementation cost: $200,000. That's a statistically significant waste of money.

Fix: Always calculate effect sizes in real units (dollars, customers, hours saved). Compare to implementation costs. Statistical significance tells you the effect is probably real. Economic significance tells you if it matters.

Pitfall 5: Assuming Causation Without Ruling Out Confounders

You see a change at the intervention point and assume you've proven causation. But did you check if competitor pricing changed that week? Did a major news event affect your industry? Did Google update their algorithm?

Fix: ITS is a quasi-experimental design, not a randomized experiment. Your causal claim rests on the assumption that nothing else changed. Document your investigation into potential confounders. Use comparison series when possible (another market/product that didn't get the intervention). Be honest about limitations.

Advanced Extensions: When the Basic Model Isn't Enough

Multiple Intervention Points

What if you rolled out the policy change in three waves (March, June, September)? You can extend the ITS model to include multiple intervention dummies and time-after-intervention terms.

Y_t = β₀ + β₁×time_t +
      β₂×intervention1_t + β₃×time_after_intervention1_t +
      β₄×intervention2_t + β₅×time_after_intervention2_t +
      β₆×intervention3_t + β₇×time_after_intervention3_t + ε_t

Each wave gets its own level and slope change estimate. This design works if the waves are staggered enough to measure each effect separately.

Controlled ITS: Adding a Comparison Series

If you have a similar market/product/region that didn't receive the intervention, use it as a comparison series to difference out shared trends. This strengthens causal inference by controlling for time-varying confounders.

Model both the treatment and control series, then test whether the intervention effect differs between them. This is called comparative interrupted time series (CITS) or difference-in-differences with time series data.

Seasonality and Trend Decomposition

If your data has strong seasonal patterns (retail sales peak in Q4), add seasonal dummy variables or use Fourier terms to model the cycle. Alternatively, decompose the series (STL decomposition) and run ITS on the deseasonalized residuals.

Non-Linear and Non-Normal Outcomes

If your outcome is a count (support tickets per day), use Poisson or negative binomial regression instead of OLS. If it's binary (policy compliance: yes/no), use logistic regression. The ITS logic stays the same: include time, intervention, and time-after-intervention predictors. Interpretation changes slightly (log-linear effects instead of linear effects).

Real-World Applications Across Industries

Healthcare: Policy Impact Evaluation

A hospital implemented a new hand hygiene protocol in April 2025. They tracked monthly hospital-acquired infection (HAI) rates for 18 months before and 12 months after. ITS analysis showed a 34% reduction in HAI rates (p < 0.001), with both level change (immediate drop) and slope change (continued decline). Estimated savings: $1.8M annually in reduced treatment costs and liability.

E-Commerce: Feature Launch Assessment

An online retailer launched product recommendations on their checkout page. They couldn't A/B test it (engineering constraints required full rollout). Using 24 months pre-launch data and 10 months post-launch, ITS detected a 7.2% increase in average order value (p = 0.009), attributing $420K in incremental revenue to the feature.

Public Policy: Regulation Effects

A city implemented a plastic bag ban in January 2025. Environmental advocates wanted evidence of impact. Using monthly landfill waste data (30 months pre, 18 months post), ITS showed a 12% reduction in plastic waste volume (p = 0.002). No change in total waste volume, suggesting substitution to reusable bags rather than overall consumption reduction.

SaaS: Pricing Strategy Change

A B2B SaaS company switched from per-seat to per-usage pricing in July 2025. Customer count was unchanged (existing contracts), so they couldn't randomize. ITS analysis on monthly recurring revenue (MRR) showed a 19% increase in MRR (p < 0.001) driven primarily by high-usage customers. ROI analysis: $2.1M incremental ARR against $150K implementation cost.

How to Report ITS Results: The Methodological Checklist

If you're publishing ITS findings (internally or externally), include these elements to meet methodological standards:

1. Design specification: Number of pre/post time points, intervention date, outcome measure
2. Visual evidence: Time series plot with intervention line clearly marked
3. Model specification: Equation, error structure (OLS vs. robust SEs vs. ARIMA), software used
4. Coefficient table: β₀, β₁, β₂, β₃ with SEs, t-stats, p-values, and 95% CIs
5. Diagnostics: Durbin-Watson stat, residual plots, autocorrelation function (ACF) plot
6. Sensitivity analyses: What happens if you exclude the first post-intervention month? Different lag structures? Robustness checks strengthen credibility.
7. Threats to validity: Documented investigation of concurrent events, confounders, and assumption violations
8. Effect size in context: Translate coefficients to real-world units and compare to costs

MCP Analytics' CSV analysis platform generates publication-ready reports with all of these elements automatically, including embedded R code so your analysis is fully reproducible.

Limitations: When ITS Isn't the Right Tool

Let's be honest about what ITS can't do:

It's Weaker Than a Randomized Experiment

If you can randomize, you should. A/B tests eliminate confounding by design. ITS relies on assumptions (no concurrent events, stable trends) that you can check but never fully prove. Use ITS when experiments aren't feasible, not as a substitute for laziness.

It Requires Good Historical Data

If you only have 4 months of pre-intervention data, or if the pre-intervention period was highly volatile, ITS won't give you reliable estimates. The method needs stable baseline trends. No amount of statistical sophistication fixes bad data.

It Can't Disentangle Multiple Simultaneous Changes

If you launched a new feature, changed pricing, and started a marketing campaign all in the same month, ITS will estimate the combined effect. It can't isolate individual contributions without additional design elements (staggered rollout, comparison groups).

It Assumes Effects Are Immediate or Gradual

The standard ITS model allows for level change (immediate) and slope change (gradual). If your intervention has a delayed onset (3-month lag before effects appear), you need to modify the model or use lag structures. This requires knowing the lag period in advance.

External Validity Is Limited

Your ITS analysis shows that a pricing change increased revenue in Q2 2025. Will it work again in Q4? In a different market? ITS measures what happened in one specific context. Generalization requires replication or meta-analysis across multiple intervention instances.

The Practical Workflow: From Data to Decision

Here's the step-by-step process I follow when running an ITS analysis:

1. Define the Intervention and Outcome (Before Looking at Data)

Write down: What changed? When exactly did it change? What outcome are you measuring? This must be pre-specified, not data-driven. Otherwise you're fishing for effects.

2. Assemble the Time Series

Collect data for as many pre-intervention time points as possible (minimum 8, target 20+). Post-intervention: at least 8 points, ideally 12+. Consistent measurement intervals (monthly, weekly).

3. Plot the Data

Visual inspection first. Do you see an obvious change at the intervention point? What's the pre-trend? Any seasonality? Outliers? Structural breaks unrelated to your intervention?

4. Fit the Segmented Regression Model

Start with basic OLS. Check Durbin-Watson for autocorrelation. If detected, refit with Newey-West SEs or Prais-Winsten transformation. Extract coefficients and p-values.

5. Run Diagnostics

Examine residual plots. Check for heteroskedasticity, non-normality, remaining autocorrelation. Plot ACF/PACF of residuals. If problems persist, adjust the model (add covariates, try non-linear terms, use ARIMA errors).

6. Sensitivity Analysis

Rerun the model excluding the first post-intervention point (in case of outliers). Try different lag structures. Check if results hold. If coefficients flip sign or significance disappears, your finding is fragile.

7. Investigate Confounders

Interview stakeholders. Check marketing calendars, competitor activity, economic indicators. If you find concurrent events, add them to the model or acknowledge as limitations.

8. Calculate Effect Sizes and ROI

Translate β₂ and β₃ into real-world units. Build the counterfactual projection. Calculate cumulative impact over the post-intervention period. Compare to costs. That's your ROI.

9. Document and Report

Write up the methods, results, diagnostics, and limitations. Include the time series plot, coefficient table, and interpretation. Be transparent about threats to validity. That's how you build credibility.

MCP Analytics' analysis creation workflow walks you through this process step-by-step, automating the statistical heavy lifting while keeping you in control of design decisions.

Software and Tools: Running Your Own ITS Analysis

If you want to run ITS manually (outside of MCP's platform), here are the tools:

R

The workhorse for ITS. Key packages:

• lmtest: Durbin-Watson test, robust standard errors
• sandwich: Newey-West HAC standard errors
• nlme: GLS models with autocorrelated errors
• forecast: ARIMA error structures
• segmented: Specialized package for segmented regression

Basic workflow in R:

library(lmtest)
library(sandwich)

# Fit model
model <- lm(outcome ~ time + intervention + time_after, data = df)

# Check autocorrelation
dwtest(model)

# Robust SEs if autocorrelation detected
coeftest(model, vcov = NeweyWest(model))

Python

Use statsmodels:

import statsmodels.api as sm
from statsmodels.regression.linear_model import OLS
from statsmodels.stats.diagnostic import acorr_breusch_godfrey

# Fit model
X = df[['time', 'intervention', 'time_after']]
y = df['outcome']
model = OLS(y, sm.add_constant(X)).fit()

# Check autocorrelation
bg_test = acorr_breusch_godfrey(model)

# Newey-West SEs
model_nw = OLS(y, sm.add_constant(X)).fit(cov_type='HAC', cov_kwds={'maxlags': 4})

Stata

* Fit ITS model
regress outcome time intervention time_after

* Newey-West SEs
newey outcome time intervention time_after, lag(4)

* Check for autocorrelation
estat bgodfrey

Or use MCP Analytics' free tool and skip the coding entirely. Upload your CSV, specify the intervention date, get results in 60 seconds with all diagnostics included.

Frequently Asked Questions

When should I use interrupted time series instead of A/B testing?

Use ITS when you cannot randomize: company-wide policy changes, regulatory interventions, market-level events, or when ethical/practical constraints prevent control groups. ITS requires pre-intervention data (minimum 8-12 time points) and assumes no other major changes occurred at the same time.

How many data points do I need before the intervention?

Minimum 8-12 pre-intervention time points to establish a stable baseline trend. More is better: 20+ points allow you to detect seasonality and validate model assumptions. The post-intervention period should be long enough to measure sustained effects (at least 8 points).

What's the difference between level change and slope change in ITS?

Level change is an immediate shift in the outcome (e.g., support tickets drop 20% overnight after deploying a new help system). Slope change is a change in the trend over time (e.g., customer acquisition costs decrease by $5/month after launching a referral program). Most interventions produce both effects.

How do I calculate ROI from an ITS analysis?

Compare the counterfactual (what would have happened without intervention) to observed outcomes. For cost savings: multiply the difference by unit cost and time periods. For revenue: use margin-adjusted lift. Account for implementation costs and discount future periods. ITS provides the causal effect estimate needed for rigorous ROI calculation.

What are the biggest threats to validity in ITS analysis?

Concurrent events (something else changed at the same time), autocorrelation (violating regression assumptions), seasonality (mistaking seasonal patterns for treatment effects), and insufficient pre-intervention data. Address these through careful design, segmented regression with proper error structures, and robustness checks.

Can I use ITS with weekly or daily data instead of monthly?

Yes, but be aware of increased autocorrelation and seasonality issues. Weekly data often shows day-of-week effects. Daily data is very noisy. You may need more sophisticated error structures (ARIMA models) or aggregation to weekly/monthly levels to get stable estimates.

What if I suspect the effect took a few months to kick in?

You can model a delay by shifting the intervention indicator. Instead of setting intervention = 1 starting March 2025, set it to 1 starting June 2025 if you think there's a 3-month lag. Alternatively, include a transition period dummy. Test different lag specifications and report sensitivity.

How do I know if my intervention effect is economically meaningful?

Statistical significance (p < 0.05) tells you the effect is probably real. Economic significance requires comparing effect size to costs. A $500/month revenue increase (p = 0.01) is statistically significant but economically meaningless if implementation cost was $500K. Always translate coefficients to dollar impacts and calculate ROI.

The Bottom Line: ITS as a Second-Best Solution

As an experimentalist, I'll always prefer a well-designed A/B test. Randomization eliminates confounding. It gives you clean causal inference without controversial assumptions.

But the real world doesn't always cooperate. You can't randomize company-wide policy changes. You can't A/B test regulatory compliance. You can't split your customer base when network effects make that impossible.

That's when interrupted time series becomes your tool. It's a quasi-experimental design with known limitations and testable assumptions. If you:

• Have sufficient pre-intervention data (8-12+ points minimum)
• Can document that no other major changes occurred concurrently
• Address autocorrelation properly
• Run sensitivity analyses and robustness checks
• Report limitations honestly

...then ITS gives you defensible causal estimates. Not as strong as a randomized trial, but far better than guessing or relying on simple before-after comparisons.

The ROI calculation framework connects statistical rigor to business impact. The segmented regression model quantifies level and slope changes. The diagnostic checks validate your assumptions. Together, these tools let you measure the impact of changes you couldn't A/B test and justify them with data CFOs will believe.

Run it properly, document it thoroughly, and you've got evidence that stands up to scrutiny. That's the standard.