Brief

A B2B software company repackaged its pricing in week 105 of the window below — a new mid-tier, published rate card, same product. Two quarters have passed. The board wants one number:

How much of the revenue since the change was caused by the change?

The naive answer is available to anyone with a spreadsheet: average the weeks after, average the weeks before, take the difference. This review shows that number, then shows what is wrong with it, then measures the effect against a counterfactual — an explicit model of the revenue the company would have booked had it changed nothing.

The gap between those two answers is roughly nineteen points of claimed growth that the pricing change did not create.

The data

One row per week. period marks the pricing change: weeks 1–104 are pre, weeks 105–130 are post.

column meaning unit
week week index across the window 1–130
week_start Monday of that week date
weekly_revenue_usd new + expansion revenue booked that week USD
period pre or post relative to the pricing change factor

The series is simulated to a stated ground truth so this document can be used to teach: a rising baseline (+80 USD/week), a real annual seasonality (±18% of base), 4% week-to-week execution noise, and a true pricing effect of exactly +15% switched on at week 105. Generation is deterministic (set.seed(20260824)); re-knitting reproduces data.csv byte-for-byte. Everything below is computed from the written-out file.

set.seed(20260824)
n   <- 130
cut <- 104          # the last pre-change week
w   <- 1:n

TRUE_EFFECT <- 0.15

baseline <- 42000 + 80 * w + 42000 * 0.18 * sin(2 * pi * (w - 10) / 52)
revenue  <- round(baseline * ifelse(w > cut, 1 + TRUE_EFFECT, 1) * (1 + rnorm(n, 0, 0.04)))

d <- data.frame(
  week               = w,
  week_start         = as.character(seq(as.Date("2024-03-04"), by = "week", length.out = n)),
  weekly_revenue_usd = revenue,
  period             = ifelse(w > cut, "post", "pre")
)
write.csv(d, "data.csv", row.names = FALSE)
table(d$period)
## 
## post  pre 
##   26  104
plot(d$week, d$weekly_revenue_usd / 1000, type = "l", lwd = 2, col = "#5fa9dd",
     xlab = "week", ylab = "weekly revenue (USD thousands)",
     main = "Weekly revenue, 130 weeks")
abline(v = cut + 0.5, col = "#F97316", lwd = 2, lty = 2)
text(cut + 0.5, min(d$weekly_revenue_usd / 1000), " pricing change", col = "#F97316", pos = 4, cex = 0.9)

The naive answer

pre_mean  <- mean(d$weekly_revenue_usd[d$period == "pre"])
post_mean <- mean(d$weekly_revenue_usd[d$period == "post"])
naive_rel <- post_mean / pre_mean - 1
knitr::kable(data.frame(
  window = c("pre (weeks 1-104)", "post (weeks 105-130)", "difference"),
  mean_weekly_revenue = c(round(pre_mean), round(post_mean), round(post_mean - pre_mean)),
  relative = c("", "", sprintf("%+.2f%%", naive_rel * 100))
), caption = "Before/after comparison of means")
Before/after comparison of means
window mean_weekly_revenue relative
pre (weeks 1-104) 46118
post (weeks 105-130) 61747
difference 15629 +33.89%

+33.9%. This is the number that gets presented, and it is wrong in a specific, knowable direction: the business was already growing, and the post window sits on a different part of the seasonal cycle than the pre window. The before/after comparison credits the pricing change with the trend and the season as well.

Building a counterfactual

The honest question is not “what happened after?” but “what would have happened anyway?” An interrupted time series answers it by fitting the pre-change behaviour, projecting it across the post window, and measuring the gap.

Attempt one: level and trend only

d$post <- as.integer(d$period == "post")
m_simple <- lm(weekly_revenue_usd ~ week + post, data = d)
cf_simple <- predict(m_simple, newdata = transform(d[d$post == 1, ], post = 0))
rel_simple <- mean(d$weekly_revenue_usd[d$post == 1] - cf_simple) / mean(cf_simple)
sprintf("%+.2f%%", rel_simple * 100)
## [1] "+22.49%"

Carrying the trend removes a third of the illusion. It still credits the pricing change with the season, because the model has no season in it.

Attempt two: level, trend and seasonality

d$s  <- sin(2 * pi * d$week / 52)
d$co <- cos(2 * pi * d$week / 52)
m_full <- lm(weekly_revenue_usd ~ week + s + co + post, data = d)
summary(m_full)$coefficients
##                Estimate Std. Error   t value      Pr(>|t|)
## (Intercept) 41667.10302 356.550998 116.86155 1.603866e-129
## week           84.77166   5.994522  14.14152  1.050880e-27
## s            2749.45592 250.424946  10.97916  4.760450e-20
## co          -7114.39897 212.166439 -33.53216  2.443623e-64
## post         8097.44476 603.927464  13.40798  5.931551e-26
cf_full  <- predict(m_full, newdata = transform(d[d$post == 1, ], post = 0))
eff_abs  <- unname(coef(m_full)["post"])
eff_ci   <- confint(m_full)["post", ]
rel_full <- eff_abs / mean(cf_full)
rel_ci   <- eff_ci / mean(cf_full)
cumulative <- sum(d$weekly_revenue_usd[d$post == 1] - cf_full)

knitr::kable(data.frame(
  measure = c("average effect per week", "relative effect", "cumulative effect, 26 weeks"),
  estimate = c(sprintf("%+.0f USD", eff_abs), sprintf("%+.2f%%", rel_full * 100),
               sprintf("%+.0f USD", cumulative)),
  ci_95 = c(sprintf("%.0f to %.0f USD", eff_ci[1], eff_ci[2]),
            sprintf("%+.2f%% to %+.2f%%", rel_ci[1] * 100, rel_ci[2] * 100), "")
), caption = "Interrupted time series with trend and seasonality")
Interrupted time series with trend and seasonality
measure estimate ci_95
average effect per week +8097 USD 6902 to 9293 USD
relative effect +15.09% +12.87% to +17.32%
cumulative effect, 26 weeks +210534 USD 
plot(d$week, d$weekly_revenue_usd / 1000, type = "l", lwd = 2, col = "#5fa9dd",
     xlab = "week", ylab = "weekly revenue (USD thousands)",
     main = "Actual against the counterfactual")
post_w <- d$week[d$post == 1]
polygon(c(post_w, rev(post_w)),
        c(d$weekly_revenue_usd[d$post == 1] / 1000, rev(cf_full / 1000)),
        col = "#F9731633", border = NA)
lines(post_w, cf_full / 1000, lwd = 2, lty = 2, col = "#333")
abline(v = cut + 0.5, col = "#F97316", lwd = 2, lty = 3)
legend("topleft", c("actual", "counterfactual (no change)"),
       col = c("#5fa9dd", "#333"), lty = c(1, 2), lwd = 2, bty = "n", cex = 0.85)

The spectrum

Each method answers a slightly different question, and the answers line up in order of how much of the world they carry.

spec <- data.frame(
  method = c("naive before/after", "trend only", "trend + seasonality", "TRUE effect (by construction)"),
  estimate = sprintf("%+.2f%%", c(naive_rel, rel_simple, rel_full, TRUE_EFFECT) * 100),
  carries = c("nothing", "growth", "growth + season", "—")
)
knitr::kable(spec, caption = "Every estimate of the same pricing change")
Every estimate of the same pricing change
method estimate carries
naive before/after +33.89% nothing
trend only +22.49% growth
trend + seasonality +15.09% growth + season
TRUE effect (by construction) +15.00% — 
vals <- c(naive_rel, rel_simple, rel_full, TRUE_EFFECT) * 100
bp <- barplot(vals, horiz = TRUE, xlim = c(0, 38),
              names.arg = c("naive", "trend", "trend+season", "TRUTH"),
              col = c("#c0392b", "#9aa0a6", "#F97316", "#3fbf6f"), las = 1,
              xlab = "estimated effect (%)")
text(vals + 1.6, bp, sprintf("%+.1f%%", vals), cex = 0.95)
abline(v = TRUE_EFFECT * 100, col = "#3fbf6f", lty = 2)

Every step of “and what else was happening anyway” moves the estimate down and toward the truth. The naive figure overstates the pricing change by more than a factor of two.

The placebo test

A counterfactual model can be talked into finding an effect that is not there. The check is to point it at a date where nothing happened. Here the pre-change period only, with a fake intervention at week 70:

pre_d <- d[d$post == 0, ]
pre_d$fake <- as.integer(pre_d$week > 70)
m_placebo <- lm(weekly_revenue_usd ~ week + s + co + fake, data = pre_d)
cf_pl  <- predict(m_placebo, newdata = transform(pre_d[pre_d$fake == 1, ], fake = 0))
pl_abs <- unname(coef(m_placebo)["fake"])
pl_ci  <- confint(m_placebo)["fake", ]
pl_p   <- summary(m_placebo)$coefficients["fake", "Pr(>|t|)"]

knitr::kable(data.frame(
  test = "fake intervention, week 70 (pre-period only)",
  estimate = sprintf("%+.0f USD/wk (%+.2f%%)", pl_abs, 100 * pl_abs / mean(cf_pl)),
  ci_95 = sprintf("%.0f to %.0f USD", pl_ci[1], pl_ci[2]),
  p = round(pl_p, 4)
), caption = "Placebo test")
Placebo test
test estimate ci_95 p
fake intervention, week 70 (pre-period only) -814 USD/wk (-1.63%) -2097 to 469 USD 0.2109

The interval spans zero and p = 0.211. The method finds nothing where nothing happened, which is the evidence that it did not simply manufacture the +15%.

What the data has to look like

standard_event_impact needs one row per period and three mapped columns: column_mapping {date: week_start, value: weekly_revenue_usd, period: period} — a date, the metric, and a marker for before/after. Minimum 15 rows.

Two constraints decide whether this analysis is possible at all:

Decision

  1. The pricing change is working, and it is worth about +8097 USD per week — 95% interval 6902 to 9293, or +15.1% against what the company would have booked otherwise. Cumulatively 210534 USD across the first 26 weeks.
  2. Retract the +34% figure wherever it has been used. It is the pricing change plus two years of growth plus a seasonal peak, and presenting it invites a forecast nobody can hit.
  3. Every future launch review runs the placebo test before the headline. A method that finds an effect at an arbitrary date cannot be trusted at the real one.