---
title: "Pricing page test: which denominator are you actually testing?"
subtitle: "Topic-05 worked example (LAT-2369) — the practitioner layer under the elementary video"
output:
  html_document:
    toc: true
    toc_float: true
    df_print: kable
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE, message = FALSE, warning = FALSE)
options(digits = 7)
```

## The business question

A B2B software company tested a **simplified pricing page** (three tiers, no feature
matrix) against the **control** page. The outcome that matters is `started_trial`: a
visitor who lands on pricing and opens a trial account.

The test ran four weeks. Weeks 1–2 split traffic evenly. From week 3 the team
**ramped the simplified page to 70% of traffic** — a normal progressive rollout, and the
reason the two arms did not end with equal exposure.

The weekly growth review looked at one number: **trials started per variant**. That
number says the simplified page won. This worked example is about why that number is
the wrong one, and what the right one does to the answer.

## The data

One row per pricing-page visitor: who, when, which page they saw, and whether they
opened a trial. This is the shape the platform's A/B tool consumes directly — row level,
not pre-aggregated, because **the denominator only exists at row level**.

The dataset is CONSTRUCTED: per-week exposure and trial counts are fixed, then row order
is shuffled with a fixed seed. Re-knitting reproduces `data.csv` byte-for-byte.

```{r generate}
set.seed(20260824)

weeks           <- sprintf("2026-W%02d", 23:26)
week_start      <- as.Date(c("2026-06-01", "2026-06-08", "2026-06-15", "2026-06-22"))

# progressive rollout: 50/50 for two weeks, then 30/70 toward the simplified page
control_visitors   <- c(275, 275, 180, 170)   # 900
simplified_visitors<- c(275, 275, 420, 430)   # 1400
control_trials     <- c( 39,  38,  25,  24)   # 126
simplified_trials  <- c( 28,  27,  42,  43)   # 140

rows <- do.call(rbind, lapply(seq_along(weeks), function(i) {
  mk <- function(variant, n_vis, n_trial) {
    data.frame(
      visit_date    = week_start[i] + rep(0:6, length.out = n_vis),
      variant       = variant,
      started_trial = c(rep(1L, n_trial), rep(0L, n_vis - n_trial)),
      stringsAsFactors = FALSE
    )
  }
  rbind(mk("control", control_visitors[i], control_trials[i]),
        mk("simplified", simplified_visitors[i], simplified_trials[i]))
}))

rows <- rows[sample(nrow(rows)), ]
rows$visitor_id <- sprintf("v_%05d", seq_len(nrow(rows)))
rows <- rows[, c("visitor_id", "visit_date", "variant", "started_trial")]
rownames(rows) <- NULL
write.csv(rows, "data.csv", row.names = FALSE)

nrow(rows)
```

```{r peek}
head(rows, 6)
```

## The weekly report the team was reading

```{r weekly}
weekly <- data.frame(
  week               = weeks,
  control_visitors   = control_visitors,
  control_trials     = control_trials,
  control_rate       = sprintf("%.1f%%", 100 * control_trials / control_visitors),
  simplified_visitors= simplified_visitors,
  simplified_trials  = simplified_trials,
  simplified_rate    = sprintf("%.1f%%", 100 * simplified_trials / simplified_visitors),
  traffic_split      = c("50 / 50", "50 / 50", "30 / 70", "28 / 72")
)
knitr::kable(weekly, align = "lrrrrrrr",
             col.names = c("week", "visitors", "trials", "rate",
                           "visitors", "trials", "rate", "split (C/S)"),
             caption = "Weekly pricing-page test. Left block: control. Right block: simplified.")
```

Read the **trials** columns down the page and the simplified page pulls ahead the moment
the ramp starts. Read the **rate** columns and it never leads in a single week.

## Two views of the same test

```{r views}
tab <- table(rows$variant, rows$started_trial)[c("control", "simplified"), c("1", "0")]
trials   <- tab[, "1"]
visitors <- rowSums(tab)
rate     <- trials / visitors

views <- data.frame(
  variant          = names(trials),
  trials_started   = as.integer(trials),
  visitors         = as.integer(visitors),
  conversion       = sprintf("%.2f%%", 100 * rate),
  row.names        = NULL
)
knitr::kable(views, align = "lrrr",
             col.names = c("variant", "trials started", "visitors", "conversion"),
             caption = "The count view and the rate view disagree about who won.")
```

**Count view:** simplified `r trials["simplified"]` vs control `r trials["control"]` —
the simplified page is ahead by `r trials["simplified"] - trials["control"]` trials.

**Rate view:** control `r sprintf("%.1f%%", 100*rate["control"])` vs simplified
`r sprintf("%.1f%%", 100*rate["simplified"])` — the control page converts better.

Both numbers are correctly computed from the same rows. They answer different questions.
The count answers *"which page produced more trials?"*, which is a question about traffic
allocation. The rate answers *"which page persuades better?"*, which is what a test is
for.

## The test

```{r proptest}
pt <- prop.test(trials, visitors)   # two-sided, Yates continuity correction (R default)
pt
diff_pp <- unname(rate["control"] - rate["simplified"]) * 100
ci_pp   <- pt$conf.int * 100
uplift  <- unname(rate["control"] / rate["simplified"])
round(c(diff_pp = diff_pp, ci_lo = ci_pp[1], ci_hi = ci_pp[2], uplift = uplift), 4)
```

```{r sensitivity}
chisq.test(tab)      # same machinery as prop.test on a 2x2
fisher.test(tab)     # exact sensitivity check
```

The control page converts `r sprintf("%.1f", diff_pp)` percentage points better, 95% CI
`r sprintf("%.2f", ci_pp[1])` to `r sprintf("%.2f", ci_pp[2])` pp, p =
`r sprintf("%.4f", pt$p.value)`. Relative, that is `r sprintf("%.2f", uplift)`× the trial
rate. **The page the count view crowned is the page that loses.**

## Charts

```{r chart-rates, fig.height=4, fig.width=7}
ci_each <- t(sapply(names(trials), function(v)
  prop.test(trials[[v]], visitors[[v]])$conf.int * 100))
par(mar = c(4, 4.5, 3, 1))
bp <- barplot(rate * 100, ylim = c(0, 20), col = c("#F97316", "#5fa9dd"),
              ylab = "trial start rate (%)", border = NA,
              names.arg = c("control", "simplified"),
              main = "Conversion with 95% confidence intervals")
arrows(bp, ci_each[, 1], bp, ci_each[, 2], angle = 90, code = 3, length = 0.08, lwd = 2)
text(bp, ci_each[, 2] + 1.4, sprintf("%.1f%%", rate * 100), font = 2)
```

```{r chart-ramp, fig.height=4, fig.width=7}
par(mar = c(4, 4.5, 3, 1))
m <- rbind(control = control_visitors, simplified = simplified_visitors)
barplot(m, beside = FALSE, col = c("#F97316", "#5fa9dd"), border = NA,
        names.arg = weeks, ylab = "pricing-page visitors",
        main = "Exposure was never equal: the rollout ramp")
legend("topleft", c("control", "simplified"), fill = c("#F97316", "#5fa9dd"),
       bty = "n", border = NA)
```

The second chart is the whole problem in one picture: from week 3 the simplified page is
served to more than twice as many visitors, so it accumulates more trials while
converting worse.

## The counter-check: does the ramp explain it?

If the rate gap were an artifact of *when* traffic was served rather than *which page*,
it would appear in the ramp weeks only. It does not.

```{r byweek}
byweek <- data.frame(
  week            = weeks,
  control_rate    = 100 * control_trials / control_visitors,
  simplified_rate = 100 * simplified_trials / simplified_visitors
)
byweek$gap_pp <- byweek$control_rate - byweek$simplified_rate
knitr::kable(byweek, digits = 2, align = "lrrr",
             col.names = c("week", "control %", "simplified %", "gap (pp)"),
             caption = "The control page leads in every week, ramp or no ramp.")
```

## The chooser the lesson teaches

1. **Which denominator, before which test.** Trials-started counts answer a question
   about traffic allocation. Conversion answers the question the test was run to settle.
   When exposure is unequal — a ramp, a staged rollout, an arm paused for a day — the
   count view stops being a weaker version of the rate view and becomes a *different and
   wrong* answer. Here it names the losing page as the winner.
2. **The test picks itself once the data is honest.** Two-proportion z and chi-square are
   the same machinery on a 2×2, and Fisher's exact agrees at these counts. The chooser
   was never which test; it was which denominator, and that is a data-shape decision.
3. **Unequal exposure is not bias, but it is not nothing.** Randomisation still holds
   per visitor, so the rate comparison is valid. What unequal exposure destroys is the
   *count* comparison — and the count is what most dashboards show.
4. **Two-sided by default.** One-sided is honest only when the direction was fixed before
   the data existed.