The business question

A B2B software company tested a simplified pricing page (three tiers, no feature matrix) against the control page. The outcome that matters is started_trial: a visitor who lands on pricing and opens a trial account.

The test ran four weeks. Weeks 1–2 split traffic evenly. From week 3 the team ramped the simplified page to 70% of traffic — a normal progressive rollout, and the reason the two arms did not end with equal exposure.

The weekly growth review looked at one number: trials started per variant. That number says the simplified page won. This worked example is about why that number is the wrong one, and what the right one does to the answer.

The data

One row per pricing-page visitor: who, when, which page they saw, and whether they opened a trial. This is the shape the platform’s A/B tool consumes directly — row level, not pre-aggregated, because the denominator only exists at row level.

The dataset is CONSTRUCTED: per-week exposure and trial counts are fixed, then row order is shuffled with a fixed seed. Re-knitting reproduces data.csv byte-for-byte.

set.seed(20260824)

weeks           <- sprintf("2026-W%02d", 23:26)
week_start      <- as.Date(c("2026-06-01", "2026-06-08", "2026-06-15", "2026-06-22"))

# progressive rollout: 50/50 for two weeks, then 30/70 toward the simplified page
control_visitors   <- c(275, 275, 180, 170)   # 900
simplified_visitors<- c(275, 275, 420, 430)   # 1400
control_trials     <- c( 39,  38,  25,  24)   # 126
simplified_trials  <- c( 28,  27,  42,  43)   # 140

rows <- do.call(rbind, lapply(seq_along(weeks), function(i) {
  mk <- function(variant, n_vis, n_trial) {
    data.frame(
      visit_date    = week_start[i] + rep(0:6, length.out = n_vis),
      variant       = variant,
      started_trial = c(rep(1L, n_trial), rep(0L, n_vis - n_trial)),
      stringsAsFactors = FALSE
    )
  }
  rbind(mk("control", control_visitors[i], control_trials[i]),
        mk("simplified", simplified_visitors[i], simplified_trials[i]))
}))

rows <- rows[sample(nrow(rows)), ]
rows$visitor_id <- sprintf("v_%05d", seq_len(nrow(rows)))
rows <- rows[, c("visitor_id", "visit_date", "variant", "started_trial")]
rownames(rows) <- NULL
write.csv(rows, "data.csv", row.names = FALSE)

nrow(rows)
## [1] 2300
head(rows, 6)
visitor_id visit_date variant started_trial
v_00001 2026-06-12 control 0
v_00002 2026-06-26 simplified 0
v_00003 2026-06-14 control 0
v_00004 2026-06-01 control 0
v_00005 2026-06-16 control 0
v_00006 2026-06-26 control 0

The weekly report the team was reading

weekly <- data.frame(
  week               = weeks,
  control_visitors   = control_visitors,
  control_trials     = control_trials,
  control_rate       = sprintf("%.1f%%", 100 * control_trials / control_visitors),
  simplified_visitors= simplified_visitors,
  simplified_trials  = simplified_trials,
  simplified_rate    = sprintf("%.1f%%", 100 * simplified_trials / simplified_visitors),
  traffic_split      = c("50 / 50", "50 / 50", "30 / 70", "28 / 72")
)
knitr::kable(weekly, align = "lrrrrrrr",
             col.names = c("week", "visitors", "trials", "rate",
                           "visitors", "trials", "rate", "split (C/S)"),
             caption = "Weekly pricing-page test. Left block: control. Right block: simplified.")
Weekly pricing-page test. Left block: control. Right block: simplified.
week visitors trials rate visitors trials rate split (C/S)
2026-W23 275 39 14.2% 275 28 10.2% 50 / 50
2026-W24 275 38 13.8% 275 27 9.8% 50 / 50
2026-W25 180 25 13.9% 420 42 10.0% 30 / 70
2026-W26 170 24 14.1% 430 43 10.0% 28 / 72

Read the trials columns down the page and the simplified page pulls ahead the moment the ramp starts. Read the rate columns and it never leads in a single week.

Two views of the same test

tab <- table(rows$variant, rows$started_trial)[c("control", "simplified"), c("1", "0")]
trials   <- tab[, "1"]
visitors <- rowSums(tab)
rate     <- trials / visitors

views <- data.frame(
  variant          = names(trials),
  trials_started   = as.integer(trials),
  visitors         = as.integer(visitors),
  conversion       = sprintf("%.2f%%", 100 * rate),
  row.names        = NULL
)
knitr::kable(views, align = "lrrr",
             col.names = c("variant", "trials started", "visitors", "conversion"),
             caption = "The count view and the rate view disagree about who won.")
The count view and the rate view disagree about who won.
variant trials started visitors conversion
control 126 900 14.00%
simplified 140 1400 10.00%

Count view: simplified 140 vs control 126 — the simplified page is ahead by 14 trials.

Rate view: control 14.0% vs simplified 10.0% — the control page converts better.

Both numbers are correctly computed from the same rows. They answer different questions. The count answers “which page produced more trials?”, which is a question about traffic allocation. The rate answers “which page persuades better?”, which is what a test is for.

The test

pt <- prop.test(trials, visitors)   # two-sided, Yates continuity correction (R default)
pt
## 
##  2-sample test for equality of proportions with continuity correction
## 
## data:  trials out of visitors
## X-squared = 8.1835, df = 1, p-value = 0.004227
## alternative hypothesis: two.sided
## 95 percent confidence interval:
##  0.01150374 0.06849626
## sample estimates:
## prop 1 prop 2 
##   0.14   0.10
diff_pp <- unname(rate["control"] - rate["simplified"]) * 100
ci_pp   <- pt$conf.int * 100
uplift  <- unname(rate["control"] / rate["simplified"])
round(c(diff_pp = diff_pp, ci_lo = ci_pp[1], ci_hi = ci_pp[2], uplift = uplift), 4)
## diff_pp   ci_lo   ci_hi  uplift 
##  4.0000  1.1504  6.8496  1.4000
chisq.test(tab)      # same machinery as prop.test on a 2x2
## 
##  Pearson's Chi-squared test with Yates' continuity correction
## 
## data:  tab
## X-squared = 8.1835, df = 1, p-value = 0.004227
fisher.test(tab)     # exact sensitivity check
## 
##  Fisher's Exact Test for Count Data
## 
## data:  tab
## p-value = 0.004021
## alternative hypothesis: true odds ratio is not equal to 1
## 95 percent confidence interval:
##  1.123106 1.909274
## sample estimates:
## odds ratio 
##   1.464819

The control page converts 4.0 percentage points better, 95% CI 1.15 to 6.85 pp, p = 0.0042. Relative, that is 1.40× the trial rate. The page the count view crowned is the page that loses.

Charts

ci_each <- t(sapply(names(trials), function(v)
  prop.test(trials[[v]], visitors[[v]])$conf.int * 100))
par(mar = c(4, 4.5, 3, 1))
bp <- barplot(rate * 100, ylim = c(0, 20), col = c("#F97316", "#5fa9dd"),
              ylab = "trial start rate (%)", border = NA,
              names.arg = c("control", "simplified"),
              main = "Conversion with 95% confidence intervals")
arrows(bp, ci_each[, 1], bp, ci_each[, 2], angle = 90, code = 3, length = 0.08, lwd = 2)
text(bp, ci_each[, 2] + 1.4, sprintf("%.1f%%", rate * 100), font = 2)

par(mar = c(4, 4.5, 3, 1))
m <- rbind(control = control_visitors, simplified = simplified_visitors)
barplot(m, beside = FALSE, col = c("#F97316", "#5fa9dd"), border = NA,
        names.arg = weeks, ylab = "pricing-page visitors",
        main = "Exposure was never equal: the rollout ramp")
legend("topleft", c("control", "simplified"), fill = c("#F97316", "#5fa9dd"),
       bty = "n", border = NA)

The second chart is the whole problem in one picture: from week 3 the simplified page is served to more than twice as many visitors, so it accumulates more trials while converting worse.

The counter-check: does the ramp explain it?

If the rate gap were an artifact of when traffic was served rather than which page, it would appear in the ramp weeks only. It does not.

byweek <- data.frame(
  week            = weeks,
  control_rate    = 100 * control_trials / control_visitors,
  simplified_rate = 100 * simplified_trials / simplified_visitors
)
byweek$gap_pp <- byweek$control_rate - byweek$simplified_rate
knitr::kable(byweek, digits = 2, align = "lrrr",
             col.names = c("week", "control %", "simplified %", "gap (pp)"),
             caption = "The control page leads in every week, ramp or no ramp.")
The control page leads in every week, ramp or no ramp.
week control % simplified % gap (pp)
2026-W23 14.18 10.18 4.00
2026-W24 13.82 9.82 4.00
2026-W25 13.89 10.00 3.89
2026-W26 14.12 10.00 4.12

The chooser the lesson teaches

  1. Which denominator, before which test. Trials-started counts answer a question about traffic allocation. Conversion answers the question the test was run to settle. When exposure is unequal — a ramp, a staged rollout, an arm paused for a day — the count view stops being a weaker version of the rate view and becomes a different and wrong answer. Here it names the losing page as the winner.
  2. The test picks itself once the data is honest. Two-proportion z and chi-square are the same machinery on a 2×2, and Fisher’s exact agrees at these counts. The chooser was never which test; it was which denominator, and that is a data-shape decision.
  3. Unequal exposure is not bias, but it is not nothing. Randomisation still holds per visitor, so the rate comparison is valid. What unequal exposure destroys is the count comparison — and the count is what most dashboards show.
  4. Two-sided by default. One-sided is honest only when the direction was fixed before the data existed.