A/B Test
Compares each variant with the control on a conversion rate or a mean, with the difference and its range, a Holm-adjusted p-value, whether traffic was split as planned, whether the test had enough rows, and how the result moved over the test.
RUN DATE · 14 September 2026
DATA · 4,795 rows
Did either new checkout design beat the current one on conversion, and can we trust the result?
- SummaryThe control, the best variant, the difference between them and how many variants beat the control.
- Outcome by variantEach variant's conversion rate or mean with its interval.
- Difference from the controlEach variant minus the control, with its interval.
- Traffic splitEach variant's share of rows beside an equal split.
- Over the testEach variant's cumulative outcome as rows arrived.
- Every comparisonEach variant against the control with its interval, relative change and adjusted p-value.
- What this test could detectRows per variant against the rows needed, and the smallest difference the test could detect.
- What the results rely onEach condition the comparison depends on, and whether it holds.
- How it was doneThe method, the data used, and what to keep in mind.
What the test found
Variant B leads, Variant C overlaps control
Variant B's conversion rises clearly above control; Variant C stays roughly level with control.
Variant B's interval sits cleanly above the other two; Variant C and Control intervals overlap substantially.
Variant B leads, Variant C overlaps control
Variant B shows a higher conversion rate than both Control and Variant C, with its interval clearly separated above them. Variant C's interval overlaps Control's interval, indicating their conversion rates are not clearly separated. All checks hold, supporting confidence in the observed differences. The key question now is whether Variant B's advantage is statistically significant and whether the sample sizes justify that conclusion.
Variant B's interval sits cleanly above the other two; Variant C and Control intervals overlap substantially.
Variant B beats control, Variant C does not
Variant B's interval sits above zero, clearly beating control; Variant C's interval crosses zero, indistinguishable from control.
Variant B's interval lies entirely above zero; Variant C's interval straddles zero on both sides.
Variant B beats control, Variant C does not
Variant B shows a clear gain over control, with its interval entirely above zero. Variant C's interval crosses zero, meaning its difference from control could be negative, zero, or positive with equal plausibility. This distinction is decisive: only Variant B has a trustworthy advantage. All five assumption checks hold, supporting the result.
Variant B's interval lies entirely above zero; Variant C's interval straddles zero on both sides.
Can the result be trusted
Variants received roughly equal traffic shares
Traffic split sits close to the equal plan across all three variants, supporting trustworthy comparison.
Observe Control, Variant C, and Variant B shares against the planned equal split line; all three cluster tightly around it.
Variants received roughly equal traffic shares
Control, Variant C, and Variant B each received shares that sit clearly close to the planned equal split, with no variant noticeably favored or starved. The tight clustering around the equal line shows the assignment worked as intended. All checks hold, so the foundation for comparing conversion across variants is sound. Next, examine whether the variants' conversion rates themselves differ meaningfully.
Observe Control, Variant C, and Variant B shares against the planned equal split line; all three cluster tightly around it.
Variant B leads clearly throughout test
Variant B separates ahead early and maintains its lead throughout, while Control and Variant C stay close behind in consistent order.
The three lines from the first checkpoint: Variant B rises well above the other two, which track closely together below it.
Variant B leads clearly throughout test
From the opening checkpoint onward, Variant B establishes a clear lead over both Control and Variant C, which remain in tight proximity to each other throughout. The separation between Variant B and the other two variants is consistent across all checkpoints, with no crossing or reordering. Control and Variant C maintain their relative positions from start to finish, with Variant C slightly ahead. The stable order and persistent gap indicate the difference is genuine and not driven by early volatility.
The three lines from the first checkpoint: Variant B rises well above the other two, which track closely together below it.
The numbers
Variant B wins, Variant C falls short
Variant B beats control clearly, Variant C does not. Only B is trustworthy.
Variant C and B diverge sharply despite similar names, with only B crossing the significance threshold.
Variant B wins, Variant C falls short
Variant B shows a significant gain over control with a tight interval that excludes zero, backed by a low p-value. Variant C's interval spans zero and its p-value is well above the test threshold, so it shows no meaningful difference. All five assumption checks hold, supporting confidence in both results.
Variant C and B diverge sharply despite similar names, with only B crossing the significance threshold.
What the test could detect
Variant B clears power bar, Variant C falls short
Variant B reaches detectable size; Variant C remains too small to trust its result.
Variant C's needed sample size vastly exceeds its actual rows, while Variant B's actual rows exceed what it needs.
Variant B clears power bar, Variant C falls short
Variant B versus Control has enough rows for the difference it observed, meeting the power threshold clearly. Variant C versus Control falls far short: its sample size would need to be much larger to detect its observed difference reliably. The test could trust Variant B's finding but not Variant C's.
Variant C's needed sample size vastly exceeds its actual rows, while Variant B's actual rows exceed what it needs.
Assumptions and method
All five checks hold
All five assumption checks hold, so nothing here limits how far the results can be trusted.
Holding: traffic split matches an equal split, enough data in every variant, test powered for the best observed difference, and two more.
All five checks hold
The checks that hold: traffic split matches an equal split, enough data in every variant, test powered for the best observed difference, outcome not dominated by extremes, rows complete.
Holding: traffic split matches an equal split, enough data in every variant, test powered for the best observed difference, and two more.
Two-proportion z-test of each variant against the control 'Control' (named like a control) over 4780 rows in 3 variants; Converted read as two values ('No', 'Yes'), success = 'Yes'; 95% intervals (Wilson for rates, Newcombe for differences in percentage points); p-values Holm-adjusted across 2 comparisons at alpha 0.05; traffic split tested against equal shares; rows needed for 80% power at the observed difference; cumulative outcome over Signup Date at 14 checkpoints; excluded: 15 rows missing a variant or outcome; not used: Device, Order Value, User ID (not mapped).
4780 of 4795 rows · Assigned Group → Converted
caveatTwo-proportion z-test against control with Holm adjustment; all five assumption checks held.
The method compared each new checkout design to the current one using a two-proportion z-test on Converted outcomes, with Holm-adjusted p-values to control false positives across both comparisons. All five assumption checks held: random assignment, row independence, planned sample size, and proper statistical interpretation. The test excluded only rows missing variant or outcome data. A reader can trust the statistical validity of the result.
The code behind this report
The code that produced every figure in this report, exactly as it ran. Fingerprint 4b072bb422ffa9fe. The same code on the same data gives the same report.
`standard_ab_test_v2` <- function(pf) {
`%||%` <- function(a, b) if (!is.null(a)) a else b
#' Readable figures (LAT-3180, LAT-3181): whole numbers from a thousand up, one decimal from a hundred, two from
#' one, three significant figures below one. A cell carries what the value needs, not what R prints.
tidy <- function(x) {
x <- as.numeric(x)
ifelse(is.na(x), NA_real_,
ifelse(abs(x) >= 1000, round(x, 0),
ifelse(abs(x) >= 100, round(x, 1),
ifelse(abs(x) >= 1, round(x, 2), signif(x, 3)))))
}
#' P-values below 0.0001 leave as text (LAT-3181): the four-decimal serializer rounds 1.4e-05 to 0.
p_text <- function(p) vapply(p, function(q) if (is.na(q)) "" else if (q < 1e-4) "<0.0001" else format(signif(q, 3)), character(1))
inputs <- pf$taskList$inputs
params <- inputs$module_parameters %||% list()
# THE QUESTION this tool answers: the customer's objective, verbatim, when given.
question <- (inputs$userContext %||% list())$objective %||%
"Did any variant beat the control, by how much, and can the difference be trusted?"
#' ## Column mapping
#' One row per visitor or unit: `variant` (the group each row saw), `outcome` (a conversion flag, a two-level yes/no
#' column, or a numeric value) and, optionally, `arrival` (when the row entered the test: a date, a timestamp or an
#' order number), which lets the report show whether the difference held over the test.
col_map <- inputs$column_mapping %||% list()
df <- renderObject.taskFunction.init(inputs, col_map) # df has SEMANTIC names
human <- function(sem) {
v <- col_map[[sem]]
if (is.null(v) || !nzchar(as.character(v))) sem else as.character(v)
}
variant_name <- human("variant"); outcome_name <- human("outcome")
for (sem in c("variant", "outcome"))
if (!(sem %in% names(df))) stop(sprintf("column_mapping must map '%s' (%s was not found).", sem, human(sem)))
has_arrival <- "arrival" %in% names(df)
n_in <- nrow(df)
#' ## The columns this tool did not look at, read from the raw rows (init narrows `df` to the mapped columns)
raw_names <- local({
ds <- inputs$dataset %||% inputs$df
if (is.data.frame(ds)) return(names(ds))
if (is.list(ds) && length(ds) > 0) {
rows <- ds[seq_len(min(length(ds), 50))]
nm <- unique(unlist(lapply(rows, function(r) if (is.list(r)) names(r) else NULL)))
if (length(nm)) return(nm)
if (!is.null(names(ds)) && all(nzchar(names(ds)))) return(names(ds))
}
character(0)
})
mapped_actual <- unique(as.character(unlist(col_map)))
ignored_cols <- setdiff(raw_names, unique(c(mapped_actual, make.names(mapped_actual))))
#' ## Parameters
#' `alpha`: the significance level every comparison is judged at, after the Holm adjustment (0.01, 0.05 or 0.10); the
#' intervals are at 1 - alpha. `control`: the variant to compare against; blank picks a variant named like a control
#' (control, baseline, original, champion, A), else the largest group.
alpha <- suppressWarnings(as.numeric(params$alpha %||% 0.05))
if (is.na(alpha) || !(alpha %in% c(0.01, 0.05, 0.10))) stop("module_parameters$alpha must be 0.01, 0.05 or 0.10")
conf <- 1 - alpha; z <- stats::qnorm(1 - alpha / 2); conf_pct <- round(100 * conf)
control_param <- trimws(as.character(params$control %||% ""))
#' ## Data preparation
#' Rows with a blank variant or outcome are excluded and counted. At most four variants are compared (the largest);
#' a variant with fewer than 5 rows is dropped and named. The outcome is a conversion when it is logical, only 0 and 1,
#' or exactly two text values (success is the one that reads like a success, else the alphabetically last); otherwise
#' it must be numeric (95% rule) and variants are compared on their means.
v_all <- trimws(as.character(df$variant)); o_raw <- df$outcome; o_chr <- trimws(as.character(o_raw))
keep <- !is.na(v_all) & v_all != "" & !is.na(o_chr) & o_chr != "" & !(tolower(o_chr) %in% c("na", "n/a", "null", "nan"))
n_missing <- sum(!keep)
arrival_raw <- if (has_arrival) df$arrival[keep] else NULL
v <- v_all[keep]; o_chr <- o_chr[keep]; o_raw <- o_raw[keep]
outcome_type <- NULL; success_how <- ""; n_uncoercible <- 0L
if (is.logical(o_raw)) {
y <- as.numeric(o_raw); outcome_type <- "binary"; success_how <- "a TRUE/FALSE column, success = TRUE"
} else {
conv <- suppressWarnings(as.numeric(o_chr))
if (sum(!is.na(conv)) >= 0.95 * length(o_chr)) {
bad <- is.na(conv); n_uncoercible <- sum(bad)
v <- v[!bad]; conv <- conv[!bad]; if (has_arrival) arrival_raw <- arrival_raw[!bad]
if (all(conv %in% c(0, 1))) { y <- conv; outcome_type <- "binary"; success_how <- "a 0/1 flag, success = 1" } else
{ y <- conv; outcome_type <- "numeric"; success_how <- "numeric, variants compared on their means" }
} else {
lv <- sort(unique(o_chr))
if (length(lv) != 2) stop(sprintf("%s has %d distinct text values; map a 0/1 conversion flag, a column with exactly two values (yes/no), or a numeric value.", outcome_name, length(lv)))
kw <- c("yes", "y", "true", "1", "converted", "success", "clicked", "purchased", "bought", "retained", "signed up", "won")
hit <- lv[tolower(lv) %in% kw]
success <- if (length(hit)) hit[1] else lv[2]
success_how <- sprintf("two values ('%s', '%s'), success = '%s'%s", lv[1], lv[2], success, if (length(hit)) "" else " (the alphabetically last; neither reads like a success)")
y <- as.numeric(o_chr == success); outcome_type <- "binary"
}
}
binary <- outcome_type == "binary"
tab <- sort(table(v), decreasing = TRUE)
top <- names(tab)[seq_len(min(4L, length(tab)))]
small <- top[as.integer(tab[top]) < 5]
kept <- setdiff(top, small); dropped_groups <- c(setdiff(names(tab), top), small)
sel <- v %in% kept; n_dropped_rows <- sum(!sel)
v <- v[sel]; y <- y[sel]; if (has_arrival) arrival_raw <- arrival_raw[sel]
if (length(kept) < 2) stop(sprintf("After cleaning, %s has %d variant(s) with 5 or more rows; an A/B test needs at least two.", variant_name, length(kept)))
if (binary && (all(y == 1) || all(y == 0))) stop(sprintf("%s is the same for every usable row; there is nothing to compare.", outcome_name))
if (!binary && isTRUE(stats::var(y) == 0)) stop(sprintf("%s has no variation; there is nothing to compare.", outcome_name))
n <- length(y); if (n < 20) stop(sprintf("Only %d usable rows; an A/B test needs at least 20.", n))
named <- kept[tolower(kept) %in% tolower(control_param)]
kw_ctrl <- kept[tolower(kept) %in% c("control", "baseline", "original", "champion", "a", "variant a", "group a", "control group")]
control <- if (nzchar(control_param) && length(named)) named[1] else if (length(kw_ctrl)) kw_ctrl[1] else kept[1]
control_how <- if (nzchar(control_param) && length(named)) "as requested" else if (nzchar(control_param)) sprintf("'%s' was requested but is not a variant, so the variant named like a control or the largest was used", control_param) else if (length(kw_ctrl)) "named like a control" else "the largest group, no variant is named like a control"
treatments <- setdiff(kept, control)
order_v <- c(control, treatments)
#' ## Per-variant outcome: Wilson interval for a rate, t interval for a mean
wilson <- function(x, m) { ph <- x / m; den <- 1 + z^2 / m; ctr <- (ph + z^2 / (2 * m)) / den
half <- z * sqrt(ph * (1 - ph) / m + z^2 / (4 * m^2)) / den; c(ctr - half, ctr + half) }
gs <- lapply(order_v, function(g) {
yy <- y[v == g]; m <- length(yy)
if (binary) { x <- sum(yy); ci <- wilson(x, m); list(n = m, x = x, est = x / m, lo = ci[1], hi = ci[2], sd = NA_real_) }
else { mu <- mean(yy); s <- stats::sd(yy); h <- stats::qt(1 - alpha / 2, m - 1) * s / sqrt(m); list(n = m, x = NA, est = mu, lo = mu - h, hi = mu + h, sd = s) }
})
names(gs) <- order_v
scale_v <- if (binary) 100 else 1
outcome_df <- data.frame(variant = order_v, value = tidy(scale_v * sapply(gs, `[[`, "est")),
ci_low = tidy(scale_v * sapply(gs, `[[`, "lo")), ci_high = tidy(scale_v * sapply(gs, `[[`, "hi")), stringsAsFactors = FALSE)
#' ## Each treatment against the control: two-proportion z-test with a Newcombe interval, or Welch's t-test
#' The results table carries the interval as one text column and only the Holm p-value: ten columns overflowed the
#' 1120px sheet by 196px (LAT-3188, run 1), and eight by 97px (run 3), so the control's value, the same on every row and
#' on the outcome chart, is left out too; the raw p-value stays in the adjustment and the answer.
cg <- gs[[control]]
comp <- lapply(treatments, function(g) {
tg <- gs[[g]]
if (binary) {
d <- tg$est - cg$est
lo <- d - sqrt((tg$est - tg$lo)^2 + (cg$hi - cg$est)^2); hi <- d + sqrt((tg$hi - tg$est)^2 + (cg$est - cg$lo)^2)
p <- tryCatch(suppressWarnings(stats::prop.test(c(tg$x, cg$x), c(tg$n, cg$n), correct = FALSE)$p.value), error = function(e) NA_real_)
rel <- if (cg$x > 0 && tg$x > 0) { se <- sqrt((1 - tg$est) / tg$x + (1 - cg$est) / cg$x); r <- tg$est / cg$est
c(r - 1, r * exp(-z * se) - 1, r * exp(z * se) - 1) } else rep(NA_real_, 3)
need <- tryCatch(if (abs(d) > 0) ceiling(stats::power.prop.test(p1 = cg$est, p2 = tg$est, sig.level = alpha, power = 0.8)$n) else NA_real_, error = function(e) NA_real_)
mde <- tryCatch(stats::power.prop.test(n = min(tg$n, cg$n), p1 = cg$est, sig.level = alpha, power = 0.8)$p2 - cg$est, error = function(e) NA_real_)
list(variant = g, d = 100 * d, lo = 100 * lo, hi = 100 * hi, p = p, rel = 100 * rel, need = need, mde = 100 * mde, nmin = min(tg$n, cg$n))
} else {
tt <- tryCatch(stats::t.test(y[v == g], y[v == control], conf.level = conf), error = function(e) NULL)
d <- tg$est - cg$est
se_r <- if (cg$est != 0) sqrt((tg$sd^2 / tg$n) / cg$est^2 + (tg$est^2 * cg$sd^2 / cg$n) / cg$est^4) else NA_real_
rel <- if (!is.na(se_r)) c(tg$est / cg$est - 1, tg$est / cg$est - 1 - z * se_r, tg$est / cg$est - 1 + z * se_r) else rep(NA_real_, 3)
sp <- sqrt((tg$sd^2 + cg$sd^2) / 2)
need <- tryCatch(if (abs(d) > 0) ceiling(stats::power.t.test(delta = abs(d), sd = sp, sig.level = alpha, power = 0.8)$n) else NA_real_, error = function(e) NA_real_)
mde <- tryCatch(stats::power.t.test(n = min(tg$n, cg$n), sd = sp, sig.level = alpha, power = 0.8)$delta, error = function(e) NA_real_)
list(variant = g, d = d, lo = if (!is.null(tt)) tt$conf.int[1] else NA_real_, hi = if (!is.null(tt)) tt$conf.int[2] else NA_real_,
p = if (!is.null(tt)) tt$p.value else NA_real_, rel = 100 * rel, need = need, mde = mde, nmin = min(tg$n, cg$n))
}
})
p_raw <- sapply(comp, `[[`, "p"); p_holm <- stats::p.adjust(p_raw, method = "holm")
sig <- !is.na(p_holm) & p_holm < alpha
labels_c <- paste(treatments, "vs", control)
diff_df <- data.frame(comparison = labels_c, difference = tidy(sapply(comp, `[[`, "d")),
low = tidy(sapply(comp, `[[`, "lo")), high = tidy(sapply(comp, `[[`, "hi")), stringsAsFactors = FALSE)
results_df <- data.frame(
comparison = labels_c, variant_value = tidy(scale_v * sapply(treatments, function(g) gs[[g]]$est)),
difference = tidy(sapply(comp, `[[`, "d")),
interval = paste(as.character(tidy(sapply(comp, `[[`, "lo"))), "to", as.character(tidy(sapply(comp, `[[`, "hi")))),
relative_pct = tidy(sapply(comp, function(r) r$rel[1])), p_holm = p_text(p_holm),
significant = ifelse(sig, "yes", "no"), stringsAsFactors = FALSE)
power_df <- data.frame(
comparison = labels_c, rows_per_variant = as.integer(sapply(comp, `[[`, "nmin")),
needed_per_variant = sapply(comp, function(r) if (is.na(r$need)) NA_real_ else r$need),
detectable_difference = tidy(sapply(comp, `[[`, "mde")), observed_difference = tidy(sapply(comp, `[[`, "d")), stringsAsFactors = FALSE)
best <- which.max(sapply(comp, `[[`, "d"))
#' ## Traffic split: observed shares against an equal split (sample ratio mismatch)
counts <- sapply(gs, `[[`, "n")
srm_p <- tryCatch(suppressWarnings(stats::chisq.test(counts, p = rep(1 / length(counts), length(counts)))$p.value), error = function(e) NA_real_)
split_df <- data.frame(variant = rep(order_v, 2), share_pct = round(c(100 * counts / sum(counts), rep(100 / length(counts), length(counts))), 1),
series = rep(c("Observed", "Planned equal split"), each = length(counts)), stringsAsFactors = FALSE)
#' ## Over the test: each variant's cumulative outcome at 20 checkpoints, when an arrival column was mapped
cum_df <- NULL; arrival_note <- if (!has_arrival) "no arrival column was mapped, so the test cannot be read over time" else NULL
if (has_arrival) {
a <- arrival_raw; kind <- NULL
if (inherits(a, "POSIXt") || inherits(a, "Date")) { t_num <- as.numeric(as.POSIXct(a)); kind <- "date" } else {
s <- trimws(as.character(a)); an <- suppressWarnings(as.numeric(s))
if (sum(!is.na(an)) >= 0.9 * length(s)) { t_num <- an; kind <- "order" } else {
fmts <- c("%Y-%m-%d %H:%M:%S", "%Y-%m-%dT%H:%M:%S", "%Y-%m-%d %H:%M", "%Y-%m-%d", "%m/%d/%Y", "%d/%m/%Y", "%Y/%m/%d")
best_t <- NULL; best_ok <- -1
for (f in fmts) { tt <- suppressWarnings(as.numeric(as.POSIXct(s, format = f, tz = "UTC"))); ok <- sum(!is.na(tt)); if (ok > best_ok) { best_ok <- ok; best_t <- tt } }
if (best_ok >= 0.9 * length(s)) { t_num <- best_t; kind <- "date" }
}
}
if (is.null(kind)) arrival_note <- sprintf("%s could not be read as dates, timestamps or order numbers for 90%% of rows, so the test cannot be read over time", human("arrival")) else {
okr <- !is.na(t_num); o <- order(t_num[okr]); vv <- v[okr][o]; yy <- y[okr][o]; tn <- t_num[okr][o]
cps <- unique(pmax(1L, round(seq(length(yy) / 20, length(yy), length.out = 20))))
lab <- if (kind == "date") format(as.Date(as.POSIXct(tn[cps], origin = "1970-01-01", tz = "UTC"))) else as.character(tidy(tn[cps]))
keep_cp <- !duplicated(lab, fromLast = TRUE); cps <- cps[keep_cp]; lab <- lab[keep_cp]
if (length(cps) >= 3) {
cum_df <- do.call(rbind, lapply(seq_along(cps), function(i) {
do.call(rbind, lapply(order_v, function(g) { sel <- vv[seq_len(cps[i])] == g
data.frame(checkpoint = lab[i], value = if (sum(sel) >= 5) tidy(scale_v * mean(yy[seq_len(cps[i])][sel])) else NA_real_, variant = g, stringsAsFactors = FALSE) }))
}))
} else arrival_note <- sprintf("%s has fewer than three distinct points in time, so the test cannot be read over time", human("arrival"))
}
}
#' ## Assumption checks (LAT-3138)
skew <- if (!binary) max(sapply(order_v, function(g) { yy <- y[v == g]; s <- stats::sd(yy); if (s == 0) 0 else abs(mean((yy - mean(yy))^3) / s^3) })) else NA_real_
min_events <- if (binary) min(sapply(gs, function(g) min(g$x, g$n - g$x))) else min(counts)
pw <- comp[[best]]; power_ratio <- if (is.na(pw$need)) NA_real_ else pw$nmin / pw$need
miss_share <- (n_missing + n_uncoercible) / n_in
checks_df <- data.frame(
check = c("Traffic split matches an equal split", "Enough data in every variant", "Test powered for the best observed difference",
"Outcome not dominated by extremes", "Rows complete"),
statistic = c("chi-square against equal shares",
if (binary) sprintf("fewest conversions or non-conversions in a variant: %d", as.integer(min_events)) else sprintf("smallest variant: %d rows", as.integer(min_events)),
if (is.na(power_ratio)) "no difference to power" else sprintf("%s of the rows 80%% power needs", paste0(round(100 * power_ratio), "%")),
if (binary) "a conversion outcome" else sprintf("largest skewness in a variant: %s", format(round(skew, 1))),
sprintf("%d of %d rows excluded", as.integer(n_missing + n_uncoercible), n_in)),
p_value = c(p_text(srm_p), "", "", "", ""),
verdict = c(if (is.na(srm_p) || srm_p >= 0.01) "holds" else if (srm_p >= 0.001) "strained" else "violated",
if (binary) { if (min_events >= 10) "holds" else if (min_events >= 5) "strained" else "violated" } else { if (min_events >= 30) "holds" else if (min_events >= 10) "strained" else "violated" },
if (is.na(power_ratio)) "strained" else if (power_ratio >= 1) "holds" else if (power_ratio >= 0.5) "strained" else "violated",
if (binary || skew <= 2) "holds" else if (skew <= 5) "strained" else "violated",
if (miss_share <= 0.05) "holds" else if (miss_share <= 0.15) "strained" else "violated"),
note = c("an unequal split the plan did not intend usually means broken assignment or tracking, which biases every comparison",
"few conversions (or few rows) make the test statistic and its interval unreliable",
"a test short of the rows it needs misses real differences and exaggerates the ones it finds",
"a few very large values can decide a difference in means; compare medians or a capped value if so",
"rows missing a variant or an outcome are left out; if they are not random, the comparison is biased"),
stringsAsFactors = FALSE)
#' ## Method, assumptions, answer
excluded <- c(if (n_missing > 0) sprintf("%d row%s missing a variant or outcome", n_missing, if (n_missing > 1) "s" else ""),
if (n_uncoercible > 0) sprintf("%d outcome value%s not read as a number", n_uncoercible, if (n_uncoercible > 1) "s" else ""),
if (length(dropped_groups)) sprintf("variant%s dropped (fewer than 5 rows, or beyond the four largest): %s", if (length(dropped_groups) > 1) "s" else "", paste(dropped_groups, collapse = ", ")))
method <- paste0(
if (binary) "Two-proportion z-test" else "Welch's t-test", " of each variant against the control '", control, "' (", control_how, ") over ",
n, " rows in ", length(order_v), " variants; ", outcome_name, " read as ", success_how, "; ", conf_pct, "% intervals (",
if (binary) "Wilson for rates, Newcombe for differences in percentage points" else "t for means and for differences", "); p-values Holm-adjusted across ",
length(treatments), " comparison", if (length(treatments) > 1) "s" else "", " at alpha ", format(alpha),
"; traffic split tested against equal shares; rows needed for 80% power at the observed difference",
if (!is.null(cum_df)) paste0("; cumulative outcome over ", human("arrival"), " at ", length(unique(cum_df$checkpoint)), " checkpoints") else "",
if (length(excluded)) paste0("; excluded: ", paste(excluded, collapse = "; ")) else "",
if (length(ignored_cols)) paste0("; not used: ", paste(utils::head(ignored_cols, 12), collapse = ", "), " (not mapped)") else "", ".")
assumptions <- list(
"Each row was assigned to its variant at random and counted once; a visitor seen in two variants breaks the comparison.",
"Rows are independent: repeated visits by one person, or clustered traffic, make the intervals too narrow.",
"The test ran to its planned size; stopping when the difference looked good inflates false positives.",
"A significant difference is a statistical one; whether it is worth shipping depends on its size and cost.",
"Several variants share one Holm adjustment, which controls the chance of any false win across them.")
answer <- list(control = control, outcome_type = outcome_type, alpha = alpha,
best_variant = treatments[best], best_difference = tidy(comp[[best]]$d), best_relative_pct = tidy(comp[[best]]$rel[1]),
best_p_holm = p_text(p_holm[best]), significant_vs_control = sum(sig), n = n)
results <- list()
#' The verdict and the headline are NOT places of a library tool (LAT-3130): the last mile writes them.
sm <- if (binary) list(variants = length(order_v), rows = n, control_rate_pct = tidy(100 * cg$est), best_rate_pct = tidy(100 * gs[[treatments[best]]]$est),
best_lift_points = tidy(comp[[best]]$d), significant_vs_control = sum(sig)) else
list(variants = length(order_v), rows = n, control_mean = tidy(cg$est), best_mean = tidy(gs[[treatments[best]]]$est),
best_difference = tidy(comp[[best]]$d), significant_vs_control = sum(sig))
results$summary_metrics <- place_metric(sm, lead = if (binary) "best_lift_points" else "best_difference", place = "summary_metrics")
results$variant_outcomes <- place_comparison(outcome_df, category = "variant", value = "value", low = "ci_low", high = "ci_high", place = "variant_outcomes")
results$difference_interval <- place_interval(diff_df, term = "comparison", value = "difference", low = "low", high = "high", place = "difference_interval")
results$traffic_split <- place_comparison(split_df, category = "variant", value = "share_pct", series = "series", place = "traffic_split")
if (!is.null(cum_df)) {
results$cumulative_outcome <- place_trend(cum_df, x = "checkpoint", y = "value", series = "variant", place = "cumulative_outcome")
} else {
results$cumulative_outcome <- place_dropped(arrival_note, place = "cumulative_outcome")
}
results$results_table <- place_table(results_df, place = "results_table")
results$power_table <- place_table(power_df, place = "power_table")
results$assumption_checks <- place_table(checks_df, place = "assumption_checks")
results$test_method <- list(kind = "metric", values = list(
method = method, n_in = n_in, n_used = n, excluded = as.list(excluded), assumptions = assumptions,
x_column = variant_name, y_column = outcome_name),
value_order = list("n_used", "n_in"))
objects <- list() # filled by the object layer, not here
list(answer = answer, method = method, n = n, results = results, objects = objects,
json_output = list(answer = answer, method = method, n = n))
}