The Verdict
Which of the 3 variants of 'Test Group' should you ship, and what does being wrong cost?
Variant B is the best arm with probability greater than 0.999 and carries an expected loss of 0.000 percentage points—the lowest risk option. It converts at 13.99% versus Control's 9.59%, a lift of 4.40 percentage points with 95% credible interval 2.78 to 6.02 pp. Shipping Variant B has an expected loss of 0.000 pp; staying on Control costs 4.400 pp. The entire credible interval sits above zero, so the direction is settled. These probabilities do not depend on how many times you looked during the test, but they cannot correct for a stopping rule you never declared, and the prior still contributes 0.067% of Variant B's posterior weight.
Analysis Overview
Beta-Binomial posterior comparison of 3 variants of 'Test Group' across 9,000 observations.
This analysis applies a Beta-Binomial conjugate model to 9,000 observations across 3 variants, deriving a posterior distribution for each variant's true conversion rate rather than a single point estimate. Each posterior is exact—no approximation—running from Beta(288.00, 2582.00) to Beta(420.00, 2714.00). The model draws 100,000 samples from each posterior to compute the probabilities a decision requires: P(beats control), P(best), and expected loss. Because the prior is Beta(1.00, 1.00) and the seed is fixed at 42, the same file always returns the same answer. Every probability carries a Monte Carlo standard error of about 0.0000, meaning reported differences beyond three decimal places are simulation noise. The control is identified by its name, 'Control', and conversions are counted as 1 in a binary field.
Data Quality
How the file was read, what was dropped, and how conversions were counted.
The short answer
The data arrived as 9,078 rows (one per visitor), with 75 dropped for missing values, leaving 9,000 observations across 3 variants and 1,009 total conversions (11.21% pooled rate). Variant D was excluded because it had fewer than 5 observations. The analysis assumes each row is an independent visitor, which cannot be verified from the file.
The detail
9,078 rows loaded with 'Converted' coded as 0 or 1; 1 was counted as conversion. 75 rows removed for missing 'Test Group' or 'Converted' values. One variant (Variant D) excluded for having fewer than 5 observations rather than pooled into another arm—pooling would invent a variant nobody ran. Final dataset: 3 variants, 9,000 observations, 1,009 conversions (11.21% pooled rate). The file read as one row per visitor, so repeat visits by the same person counted as separate rows would make credible intervals narrower than they should be.
What this can't tell you
The assumption of independence cannot be verified from the data. If the same visitor appears multiple times and is counted as separate rows, every interval reported here is overly narrow. The file does not record timestamps, so whether the test was monitored cannot be assessed from these rows alone.
Posterior Rates by Variant
Observed and posterior conversion rate per variant, each with a 95% credible interval.
| Variant | Trials | Conversions | Observed Rate PCT | Posterior Rate PCT | CI Low PCT | CI High PCT |
|---|---|---|---|---|---|---|
| Control | 3000 | 287 | 9.567 | 9.594 | 8.566 | 10.67 |
| Variant B | 3000 | 419 | 13.97 | 13.99 | 12.77 | 15.25 |
| Variant C | 3000 | 303 | 10.1 | 10.13 | 9.073 | 11.23 |
Variant B's posterior rate of 13.991% is the highest, pinned down with a 95% credible interval of 12.773% to 15.254%—a width of 2.48 percentage points on 3,000 observations. Control's interval is tightest at 8.566% to 10.672% (width 2.11 pp), while Variant C sits at 10.127% (9.073% to 11.23%). The posterior rates sit slightly closer together than the raw rates because the Beta(1.00, 1.00) prior pulls every arm toward the middle, shrinking small arms so a 1-for-2 arm does not read as 50%. Variant B's observed rate of 13.967% is 419 conversions from 3,000 trials; the posterior shifts it marginally to 13.991%, a negligible move that reflects the prior's minimal influence on well-powered arms.
Conversion Rate by Variant
Posterior conversion rate per variant with 95% credible intervals.
Variant B's posterior rate of 13.991% stands clearly above Control's 9.594% and Variant C's 10.127%. Its 95% credible interval (12.773% to 15.254%) overlaps neither Control nor Variant C, making the separation visually decisive. Variant C's interval (9.073% to 11.23%) overlaps Control's (8.566% to 10.672%), indicating they have not cleanly separated. The credible intervals reflect sample size: all three variants drew 3,000 observations each, so their whisker widths are comparable. Overlap is a rough visual guide; the draw-by-draw probabilities in the decision table provide the true comparison—Variant C beats Control with probability 0.7561, confirming the visual overlap corresponds to genuine uncertainty.
Posterior Lift vs Control
The full posterior distribution of the difference in conversion rate, in percentage points.
The posterior lift of Variant B over Control centers at 4.40 percentage points, with 95% of the mass between 2.78 and 6.02 pp. Greater than 0.999 of the posterior lies above zero, so the direction is certain. The interval spans 3.25 percentage points, which means a decision that would flip within that range cannot be made from this data alone. The bulk of the distribution sits in the 2.5–3.0 pp range, with the tail extending upward. Because the entire credible interval is above zero, you can rule out the possibility that Variant B is actually worse than Control; what remains uncertain is how much better it is.
Decision Table
Probability of beating the control, probability of being best, and the expected loss of shipping each variant.
| Variant | Prob Beats Control | Prob Best | Expected Loss Pp | Lift Pp | Lift CI Low Pp | Lift CI High Pp | Decision Note |
|---|---|---|---|---|---|---|---|
| Control | — | 0 | 4.4 | 0 | — | — | The control. Shipping nothing costs an expected 4.400 percentage points of conversion rate against the best arm. |
| Variant B | 1 | 1 | 0 | 4.397 | 2.777 | 6.023 | Clears the conventional 0.95 bar for being best; the expected cost of shipping it is 0.000 percentage points. |
| Variant C | 0.7561 | 0 | 3.867 | 0.533 | -0.979 | 2.048 | Not separated from the control: P(beats control) = 0.756, so the data has not decided this comparison. |
Variant B beats Control with probability 1.0000 and is best with probability 1.0000, carrying an expected loss of 0.000 percentage points—the lowest risk of all three arms. Its lift is 4.397 pp with credible interval 2.777 to 6.023 pp. Control, the baseline, has an expected loss of 4.3997 pp if you ship nothing. Variant C beats Control with probability 0.7561 but is never best, with expected loss 3.8669 pp. The 0.95 bar for P(best) is a convention; Variant B clears it decisively. The expected-loss column is what a decision should turn on, because it is denominated in conversion rate—the thing you care about. Shipping Variant B costs 0.000 pp; staying on Control costs 4.400 pp.
Priors, Peeking, and What This Cannot Tell You
The prior's influence, a sensitivity check across three priors, and an honest account of the stopping-rule problem.
| Prior | Leader Under Prior | Prob Leader Is Best | Leader Rate PCT | Expected Loss Pp |
|---|---|---|---|---|
| Uniform Beta(1.00, 1.00) — the prior used above | Variant B | 1 | 13.99 | 0 |
| Jeffreys Beta(0.50, 0.50) — the standard reference prior | Variant B | 1 | 13.98 | 0 |
| Skeptical Beta centred on the pooled rate 11.21%, worth 50 pseudo-trials per variant | Variant B | 1 | 13.92 | 0 |
The short answer
The prior contributes only 0.067% of Variant B's 3,000 observations, so the data dominates. Across three different priors (uniform, Jeffreys, and skeptical), Variant B remains the leader with P(best) at 1.0000, and expected loss stays at 0.000 pp. The posterior depends only on the totals (1,009 conversions out of 9,000), not on how many times you looked while the test ran—but that is not a licence to stop whenever the number looks good.
The detail
Prior sensitivity: uniform Beta(1.00, 1.00) yields Variant B at 13.993% with P(best) = 1.0000 and expected loss = 0.000 pp; Jeffreys Beta(0.50, 0.50) yields 13.981% with P(best) = 1.0000 and expected loss = 0.000 pp; skeptical Beta centred on 11.21% (50 pseudo-trials per variant) yields 13.923% with P(best) = 1.0000 and expected loss = 0.000 pp. The Monte Carlo standard error on Variant B's P(best) of 1.0000 is 0.0000, so the third decimal place is simulation noise.
What this can't tell you
If you monitored the test and stopped the first time P(best) crossed a bar, the reported greater than 0.999 overstates how often that decision rule would be right, and nothing in this data can measure by how much. The posterior is not invalidated by monitoring, but a rule that stops the first time a threshold is crossed is still optimistic. The fix is to set the rule (the bar and minimum sample) before you look, not to reinterpret the number afterwards.
Methods & Disclosure
The model, the prior, the draws, the control, and the limits of what this analysis can decide.
| Item | Detail |
|---|---|
| Model | Beta-Binomial conjugate: each variant's conversion rate gets a Beta posterior with parameters (prior + conversions, prior + non-conversions). No model is fitted and no normal approximation is used, so the arms with few observations are handled correctly. |
| Prior | Beta(1.00, 1.00), uniform over the conversion rate. Posterior parameters run from Beta(288.00, 2582.00) to Beta(420.00, 2714.00) across the 3 variants. |
| Posterior draws | 100,000 draws per variant from each posterior, seed 42, so re-running this analysis on the same file returns the same probabilities. |
| Monte Carlo precision | The Monte Carlo standard error on Variant B's P(best) of 1.0000 is 0.0000, so the third decimal place is noise from the simulation rather than signal from the data. |
| Control group | 'Control' (its name identifies it as the control). Every lift, credible interval and P(beats control) in this report is measured against it. |
| Data shape | One row per visitor or session, aggregated to per-variant totals by this analysis. 'Converted' holds only 0 and 1; 1 was counted as a conversion. |
| How conversions were counted | 1,009 conversions out of 9,000 observations overall, a pooled rate of 11.21%. |
| Does monitoring invalidate this? | The posterior depends only on the totals — 1,009 conversions out of 9,000 observations — and not on how many times you looked while the test ran. That is a real advantage over a fixed-horizon p-value, which is only valid for the sample size you committed to in advance. It is NOT a licence to stop whenever the number looks good. |
| What this cannot correct for | This analysis was run once, on a file that does not record when you looked. If you monitored the test and stopped the first time P(best) crossed a bar, the reported greater than 0.999 for Variant B overstates how often that decision rule would be right, and nothing in this data can measure by how much. The fix is to fix the rule — the bar, and the minimum sample — before you look, not to reinterpret the number afterwards. |
| Who sets the threshold | The expected loss of shipping Variant B is 0.000 percentage points of conversion rate. Whether that is small enough to act on is a business judgement about what a percentage point is worth to you — the analysis computes the risk, it cannot set the threshold. |
The short answer
Each variant's conversion rate posterior is Beta(prior + conversions, prior + non-conversions)—for Variant B, Beta(420.00, 2582.00). The prior is Beta(1.00, 1.00), uniform; posterior probabilities are computed from 100,000 draws per variant with seed 42 fixed, so the same file always returns the same answer. The analysis computes the risk of shipping each variant but cannot set the threshold at which that risk becomes acceptable.
The detail
Model: Beta-Binomial conjugate, exact (no normal approximation). Posterior parameters: Control Beta(288.00, 2582.00), Variant B Beta(420.00, 2714.00), Variant C Beta(304.00, 2698.00). Prior: Beta(1.00, 1.00), uniform, contributing 2.00 pseudo-observations per variant. Posterior draws: 100,000 per variant, seed 42. Control identified by name 'Control'; conversions coded as 1 in 'Converted' column. Data shape: one row per visitor, aggregated to per-variant totals. Total: 1,009 conversions out of 9,000 observations (11.21% pooled rate).
What this can't tell you
The analysis cannot correct for a stopping rule it was never told about. If you monitored and stopped when P(best) crossed a bar, the reported probability overstates how often that decision rule would succeed. The expected loss of shipping Variant B is 0.000 percentage points; whether that is small enough to act on is a business judgement about what a percentage point is worth—the analysis computes the risk, it cannot set the threshold.
Methodology
Statistical methodology and diagnostics for Bayesian A/B Test
Statistical Method
Standard-library analysis: the Bayesian read on your experiment. Upload one row per visitor with the variant they saw and whether they converted, and get a Beta posterior for every variant's true conversion rate — the probability each variant beats the control, the probability each is the best of all, the expected loss (risk) of shipping each one, and the full posterior distribution of the lift rather than a single number. Because the posterior depends only on the totals, it does not go stale the way a fixed-horizon p-value does when you check the test while it runs — and the report says plainly what that does and does not protect you from, with the prior's influence measured against your own sample sizes. Already have per-variant totals? Map a trial-count column too and it reads pre-aggregated rows instead.
- Each row is one independent visitor or session seeing one variant (or, in pre-aggregated mode, each row is one variant's totals)
- Variants were assigned randomly, so the arms are comparable on everything except the variant
- The outcome is a conversion — something that either happened or did not — not a numeric amount
- Conversions within a variant are exchangeable, which is what makes the Binomial likelihood the right one
- The prior is a genuine input, not a neutral one — the report quantifies its weight against your sample size and re-runs the conclusion under three priors, but with very few trials per arm the prior does real work
- The posterior is not invalidated by monitoring a running test, but a stopping rule tuned after the fact is still optimistic, and the analysis cannot correct for a rule it was never told about
- Probabilities are estimated by simulation, so they carry a Monte Carlo standard error that the report states — the third decimal place is usually noise
- Repeated visits by the same person counted as separate rows break independence and make every credible interval narrower than it should be; nothing in the file reveals this
Analysis Code
Complete R source code for this analysis
Bayesian A/B Test — Beta-Binomial Conjugate Analysis
Answers the question a frequentist significance test cannot: given what you have seen, what is the probability each variant is the best one, and what does it cost you to be wrong? Each variant's conversion rate gets a Beta posterior (conjugate to the Binomial likelihood); posterior draws then give P(variant beats control), P(variant is best), the expected loss (risk) of shipping each variant, and credible intervals on the lift.
Why This Method?
A p-value answers "how surprising would this data be if nothing were happening?" — a question nobody asked. The posterior answers the question people actually ask: how likely is B better than A, by how much, and what do I lose if I pick wrong. Because the Beta prior is conjugate to the Binomial likelihood, the posterior has a closed form and needs no fitting.
What This Analysis Covers
- Posterior conversion rate per variant with a 95% credible interval
- P(variant beats control) and P(variant is the best of all)
- Expected loss (risk) of shipping each variant
- The full posterior distribution of the lift, not just a point estimate
- An honest treatment of the "no peeking problem": what the posterior
really is and is not immune to, with the prior's influence quantified from the user's own sample sizes
Standard Library
Platform standard-library module (LAT-1441): runs on ANY dataset via the semantic mapping {variant, outcome, trials}. All narrative is derived from the user's own column names and computed values.
suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))Core Analysis Pipeline
compute_shared <- function(df, params, col_map = list()) {
# === SHARED EXPORTS ===
# initial_rows/final_rows/rows_removed $ row accounting
# variant_h / outcome_h / trials_h $ humanized user column names
# shape / shape_label $ "row-per-unit" | "summary" + prose label
# success_how $ how the success level was determined (prose)
# variants $ character — kept variant labels (control first)
# trials / conv $ integer — per kept variant
# dropped_levels / drop_note $ variants excluded + why
# n_missing_rows $ rows dropped for a missing variant/outcome
# a0 / b0 $ the prior's Beta parameters
# post_a / post_b $ posterior Beta parameters per variant
# post_mean / ci_lo / ci_hi $ posterior rate + 95% credible interval
# control / control_how $ control label + how it was chosen
# p_beat / p_best / e_loss $ decision quantities per variant
# lift / lift_lo / lift_hi $ absolute lift vs control (proportion units)
# rel_lift / rel_lo / rel_hi $ relative lift vs control (fraction)
# leader / focus $ highest P(best) / best non-control variant
# decisive / separated $ leader clears 0.95 / clears the runner-up
# mc_se_leader $ Monte Carlo standard error on the leader's P(best)
# prior_weight_pct $ % of the leader's posterior weight from the prior
# sens_df $ P(best) for the leader under 3 priors
# posteriors_df / rates_df / lift_draws_df / decision_df / disclosure_df
# metrics / json_output
# === /SHARED EXPORTS ===
N_DRAWS <- 100000L # posterior draws per variant (seed fixed below)
MAX_VARIANTS <- 8L # keep at most this many arms
MIN_TRIALS_PER_VARIANT <- 5L
MIN_TOTAL_TRIALS <- 20L
SKEPTICAL_WEIGHT <- 50 # pseudo-trials per variant in the skeptical prior
variant_h <- humanize_semantic("variant", col_map)
outcome_h <- humanize_semantic("outcome", col_map)
trials_h <- humanize_semantic("trials", col_map)Step 1: Validate the mapping
initial_rows <- nrow(df)
if (!("variant" %in% names(df))) {
stop(sprintf("This analysis needs a variant column mapped — the column that says which version each row belongs to(mapped as '%s').",
variant_h))
}
if (!("outcome" %in% names(df))) {
stop(sprintf("This analysis needs an outcome column mapped — the conversion flag, or the conversion COUNT if your rows are already totalled by variant(mapped as '%s').",
outcome_h))
}Step 2: Which shape is this — one row per unit, or pre-aggregated totals?
The widget maps two columns (variant + outcome) and treats every row as one visitor. Mapping a third column of trial counts switches the module to summary mode, where each row is a whole variant.
shape <- if ("trials" %in% names(df)) "summary" else "row-per-unit"
v_all <- trimws(as.character(df$variant))
v_ok <- !is.na(v_all) & v_all != ""
success_how <- ""
if (shape == "summary") {Summary shape: outcome = conversions, trials = trials, one row per variant.
coerce_count <- function(v, label_h) {
ch <- trimws(as.character(v))
non_blank <- !is.na(ch) & ch != ""
conv <- suppressWarnings(as.numeric(ch))
if (sum(non_blank) == 0 ||
sum(!is.na(conv[non_blank])) < 0.95 * sum(non_blank)) {
stop(sprintf("The column '%s' does not look numeric — fewer than 95%% of its values parse as numbers. In summary mode it must hold whole counts.",
label_h))
}
conv
}
tr_all <- coerce_count(df$trials, trials_h)
cv_all <- coerce_count(df$outcome, outcome_h)
ok <- v_ok & !is.na(tr_all) & !is.na(cv_all)
n_missing_rows <- sum(!ok)
vv <- v_all[ok]; tt <- tr_all[ok]; cc <- cv_all[ok]
if (length(vv) == 0) {
stop(sprintf("No rows have a usable '%s', '%s' and '%s' together.",
variant_h, outcome_h, trials_h))
}
if (any(tt < 0) || any(cc < 0)) {
stop(sprintf("'%s' and '%s' must be counts of zero or more — negative values were found.",
outcome_h, trials_h))
}
bad <- which(cc > tt)
if (length(bad) > 0) {
stop(sprintf("'%s' exceeds '%s' for %d row(s) (for example %s: %s conversions out of %s). Conversions cannot outnumber the people who saw the variant.",
outcome_h, trials_h, length(bad), vv[bad[1]],
format(cc[bad[1]]), format(tt[bad[1]])))
}
agg_tr <- tapply(tt, vv, sum)
agg_cv <- tapply(cc, vv, sum)
success_how <- sprintf("Each row of your file is a whole variant: '%s' is the number of people who saw it and '%s' is the number who converted.",
trials_h, outcome_h)
} else {Row-per-unit shape: every row is one visitor; the outcome is a flag.
o_all <- df$outcome
o_is_logical <- is.logical(o_all)
o_chr_all <- trimws(as.character(o_all))
ok <- v_ok & !is.na(o_all) & !is.na(o_chr_all) & o_chr_all != ""
n_missing_rows <- sum(!ok)
vv <- v_all[ok]
o_chr <- o_chr_all[ok]
if (length(vv) == 0) {
stop(sprintf("No rows have both a '%s' and an '%s' value.", variant_h, outcome_h))
}
y <- NULL
if (o_is_logical) {
y <- as.numeric(o_all[ok])
success_how <- sprintf("'%s' is a TRUE/FALSE column; TRUE was counted as a conversion.", outcome_h)
} else {
num <- suppressWarnings(as.numeric(o_chr))
if (sum(!is.na(num)) >= 0.95 * length(o_chr)) {
keep_num <- !is.na(num)
vv <- vv[keep_num]; num <- num[keep_num]
n_missing_rows <- n_missing_rows + sum(!keep_num)
uu <- sort(unique(num))
if (!all(uu %in% c(0, 1))) {
stop(sprintf("'%s' holds %d distinct numeric values (for example %s and %s). A Bayesian A/B test models a CONVERSION RATE, so it needs a 0/1 flag per row — or map a trial-count column as well and give it conversion counts per variant.",
outcome_h, length(uu), f2(uu[1]), f2(uu[length(uu)])))
}
y <- num
success_how <- sprintf("'%s' holds only 0 and 1; 1 was counted as a conversion.", outcome_h)
} else {
lv <- sort(unique(o_chr))
if (length(lv) != 2) {
stop(sprintf("'%s' has %d distinct non-numeric values, so it cannot be read as a conversion flag. Map a 0/1 column, a two-level yes/no column, or add a trial-count column and supply conversion counts.",
outcome_h, length(lv)))
}
kw <- c("yes", "y", "true", "t", "converted", "conversion", "success",
"won", "win", "clicked", "purchased", "paid", "subscribed",
"signup", "signed up", "1")
hit <- lv[tolower(lv) %in% kw]
if (length(hit) >= 1) {
success_label <- hit[1]
success_how <- sprintf("'%s' has two values ('%s' and '%s'); '%s' was counted as a conversion because it matches a known success word.",
outcome_h, lv[1], lv[2], success_label)
} else {
success_label <- lv[2]
success_how <- sprintf("'%s' has two values ('%s' and '%s') and neither matches a known success word, so '%s' — the alphabetically later one — was counted as the conversion. Check that this is the direction you meant.",
outcome_h, lv[1], lv[2], success_label)
}
y <- as.numeric(o_chr == success_label)
}
}
agg_tr <- tapply(rep(1, length(y)), vv, sum)
agg_cv <- tapply(y, vv, sum)
}Step 3: Minimum-size guards, named in the user's own columns
all_levels <- names(agg_tr)
trials_all <- as.numeric(agg_tr[all_levels])
conv_all <- as.numeric(agg_cv[all_levels])
total_trials_all <- sum(trials_all)
if (total_trials_all < MIN_TOTAL_TRIALS) {
stop(sprintf("Only %s observation(s) of '%s' are usable — a Bayesian A/B test needs at least %d in total, and at least %d per group of '%s'.",
format(total_trials_all, big.mark = ","), variant_h,
MIN_TOTAL_TRIALS, MIN_TRIALS_PER_VARIANT, variant_h))
}Keep the largest arms; drop tiny ones rather than pooling them (pooling distinct variants into an "Other" bucket would invent a variant nobody ran).
ord <- order(-trials_all, all_levels)
ranked <- all_levels[ord]
beyond_cap <- if (length(ranked) > MAX_VARIANTS) ranked[(MAX_VARIANTS + 1):length(ranked)] else character(0)
candidates <- setdiff(ranked, beyond_cap)
too_small <- candidates[as.numeric(agg_tr[candidates]) < MIN_TRIALS_PER_VARIANT]
variants <- setdiff(candidates, too_small)
dropped_levels <- c(beyond_cap, too_small)
if (length(variants) < 2) {
stop(sprintf("After cleaning, '%s' has %d group(s) with at least %d observation(s) — a comparison needs at least two.",
variant_h, length(variants), MIN_TRIALS_PER_VARIANT))
}
trials <- as.numeric(agg_tr[variants])
conv <- as.numeric(agg_cv[variants])
if (sum(conv) == 0) {
stop(sprintf("Not one of the %s observations converted — every value of '%s' is a non-conversion, so there is no conversion rate to compare.",
format(sum(trials), big.mark = ","), outcome_h))
}
final_rows <- if (shape == "summary") length(variants) else sum(trials)
rows_removed <- initial_rows - final_rowsStep 4: Choose the control — a group named like one, else the largest
named_ctrl <- variants[tolower(variants) %in%
c("control", "a", "baseline", "original", "ctrl", "holdout")]
control <- if (length(named_ctrl) > 0) named_ctrl[1] else variants[safe_which_max(trials)]
control_how <- if (length(named_ctrl) > 0) {
sprintf("'%s' (its name identifies it as the control)", control)
} else {
sprintf("'%s' (the largest group, since no group is named like a control)", control)
}Put the control first so every table reads control-then-challengers.
variants <- c(control, setdiff(variants, control))
trials <- as.numeric(agg_tr[variants])
conv <- as.numeric(agg_cv[variants])
k <- length(variants)
ctrl_i <- 1LStep 5: Conjugate posteriors — Beta(a0 + conversions, b0 + failures)
a0 <- 1; b0 <- 1 # uniform prior over the conversion rate
post_a <- a0 + conv
post_b <- b0 + (trials - conv)
post_mean <- post_a / (post_a + post_b)
ci_lo <- stats::qbeta(0.025, post_a, post_b)
ci_hi <- stats::qbeta(0.975, post_a, post_b)
obs_rate <- conv / trialsStep 6: Posterior draws — every decision quantity comes from these
set.seed(42)
draw_matrix <- function(aa, bb) {
matrix(stats::rbeta(N_DRAWS * length(aa),
rep(aa, each = N_DRAWS),
rep(bb, each = N_DRAWS)),
nrow = N_DRAWS, ncol = length(aa))
}
D <- draw_matrix(post_a, post_b)
best_idx <- max.col(D, ties.method = "first")
row_max <- D[cbind(seq_len(N_DRAWS), best_idx)]
p_best <- as.numeric(tabulate(best_idx, nbins = k)) / N_DRAWS
e_loss <- colMeans(row_max - D)
ctrl_draws <- D[, ctrl_i]
p_beat <- colMeans(D > ctrl_draws)
p_beat[ctrl_i] <- NA_real_
lift <- post_mean - post_mean[ctrl_i] # exact posterior mean of the difference
lift_lo <- rep(NA_real_, k); lift_hi <- rep(NA_real_, k)
rel_lift <- rep(NA_real_, k); rel_lo <- rep(NA_real_, k); rel_hi <- rep(NA_real_, k)
for (i in seq_len(k)) {
if (i == ctrl_i) next
dl <- D[, i] - ctrl_draws
q <- stats::quantile(dl, c(0.025, 0.975), names = FALSE)
lift_lo[i] <- q[1]; lift_hi[i] <- q[2]
rl <- dl / ctrl_draws
qr <- stats::quantile(rl, c(0.025, 0.5, 0.975), names = FALSE)
rel_lo[i] <- qr[1]; rel_lift[i] <- qr[2]; rel_hi[i] <- qr[3]
}Step 7: Who leads, and is the lead real enough to act on?
lead_i <- safe_which_max(p_best)
leader <- variants[lead_i]
p_best_leader <- p_best[lead_i]
runner_i <- safe_which_max(replace(p_best, lead_i, NA_real_))
p_best_runner <- p_best[runner_i]
mc_se_leader <- sqrt(p_best_leader * (1 - p_best_leader) / N_DRAWS)
decisive <- p_best_leader >= 0.95
separated <- (p_best_leader - p_best_runner) > 0.05The reported lift picture always compares a challenger with the control.
non_ctrl <- setdiff(seq_len(k), ctrl_i)
focus_i <- non_ctrl[safe_which_max(p_best[non_ctrl])]
focus <- variants[focus_i]Step 8: How much of the answer is the prior, not the data?
prior_weight_pct <- 100 * (a0 + b0) / (a0 + b0 + trials[lead_i])
pooled_rate <- sum(conv) / sum(trials)
prior_defs <- list(
list(label = sprintf("Uniform Beta(%s, %s) — the prior used above", f2(a0), f2(b0)),
a = rep(a0, k), b = rep(b0, k)),
list(label = "Jeffreys Beta(0.50, 0.50) — the standard reference prior",
a = rep(0.5, k), b = rep(0.5, k)),
list(label = sprintf("Skeptical Beta centred on the pooled rate %s%%, worth %s pseudo-trials per variant",
pctf(pooled_rate), format(SKEPTICAL_WEIGHT)),
a = rep(SKEPTICAL_WEIGHT * pooled_rate, k),
b = rep(SKEPTICAL_WEIGHT * (1 - pooled_rate), k))
)
sens_rows <- list()
for (pd in prior_defs) {
set.seed(42)
Ds <- draw_matrix(pd$a + conv, pd$b + (trials - conv))
bi <- max.col(Ds, ties.method = "first")
pb <- as.numeric(tabulate(bi, nbins = k)) / N_DRAWS
rm_s <- Ds[cbind(seq_len(N_DRAWS), bi)]
el <- colMeans(rm_s - Ds)
sens_rows[[length(sens_rows) + 1]] <- data.frame(
prior = pd$label,
leader_under_prior = variants[safe_which_max(pb)],
prob_leader_is_best = round(max(pb), 4),
leader_rate_pct = round(100 * mean(Ds[, safe_which_max(pb)]), 3),
expected_loss_pp = round(100 * el[safe_which_max(pb)], 4),
stringsAsFactors = FALSE
)
}
sens_df <- do.call(rbind, sens_rows)
rownames(sens_df) <- NULL
sens_spread <- max(sens_df$prob_leader_is_best) - min(sens_df$prob_leader_is_best)
sens_agree <- length(unique(sens_df$leader_under_prior)) == 1Step 10: Disclosure — what the posterior is and is not immune to
peek_line <- paste0(
"The posterior depends only on the totals — ", format(sum(conv), big.mark = ","),
" conversions out of ", format(sum(trials), big.mark = ","),
" observations — and not on how many times you looked while the test ran. ",
"That is a real advantage over a fixed-horizon p-value, which is only valid ",
"for the sample size you committed to in advance. It is NOT a licence to stop ",
"whenever the number looks good."
)
stop_rule_line <- paste0(
"This analysis was run once, on a file that does not record when you looked. ",
"If you monitored the test and stopped the first time P(best) crossed a bar, ",
"the reported ", fmt_prob(p_best_leader), " for ", leader,
" overstates how often that decision rule would be right, and nothing in this ",
"data can measure by how much. The fix is to fix the rule — the bar, and the ",
"minimum sample — before you look, not to reinterpret the number afterwards."
)
prior_line <- paste0(
"The prior adds ", f2(a0 + b0), " pseudo-observations to each variant, which is ",
f3(prior_weight_pct), "% of ", leader, "'s ", format(trials[lead_i], big.mark = ","),
" observations. Under the three priors tested, P(best) for the leading variant spans ",
f3_min(sens_spread), " and the leading variant ",
if (sens_agree) "does not change" else "CHANGES, so the conclusion is prior-driven, not data-driven",
"."
)
threshold_line <- paste0(
"The expected loss of shipping ", leader, " is ", ppf_min(e_loss[lead_i], 3),
" percentage points of conversion rate. Whether that is small enough to act on ",
"is a business judgement about what a percentage point is worth to you — the ",
"analysis computes the risk, it cannot set the threshold."
)
disclosure_df <- data.frame(
item = c("Model", "Prior", "Posterior draws", "Monte Carlo precision",
"Control group", "Data shape", "How conversions were counted",
"Does monitoring invalidate this?", "What this cannot correct for",
"Who sets the threshold"),
detail = c(
sprintf("Beta-Binomial conjugate: each variant's conversion rate gets a Beta posterior with parameters (prior + conversions, prior + non-conversions). No model is fitted and no normal approximation is used, so the arms with few observations are handled correctly.")
,
sprintf("Beta(%s, %s), uniform over the conversion rate. Posterior parameters run from Beta(%s, %s) to Beta(%s, %s) across the %d variants.",
f2(a0), f2(b0), f2(min(post_a)), f2(min(post_b)),
f2(max(post_a)), f2(max(post_b)), k),
sprintf("%s draws per variant from each posterior, seed 42, so re-running this analysis on the same file returns the same probabilities.",
format(N_DRAWS, big.mark = ",")),
sprintf("The Monte Carlo standard error on %s's P(best) of %s is %s, so the third decimal place is noise from the simulation rather than signal from the data.",
leader, f4(p_best_leader), f4(mc_se_leader)),
sprintf("%s. Every lift, credible interval and P(beats control) in this report is measured against it.", control_how),
sprintf("%s %s", if (shape == "summary")
"Pre-aggregated totals: one row per variant." else
"One row per visitor or session, aggregated to per-variant totals by this analysis.",
success_how),
sprintf("%s conversions out of %s observations overall, a pooled rate of %s%%.",
format(sum(conv), big.mark = ","), format(sum(trials), big.mark = ","),
pctf(pooled_rate)),
peek_line,
stop_rule_line,
threshold_line
),
stringsAsFactors = FALSE
)
drop_note <- if (length(dropped_levels) > 0) {
sprintf("%d group(s) of '%s' were excluded (fewer than %d observations, or beyond the %d largest arms): %s. ",
length(dropped_levels), variant_h, MIN_TRIALS_PER_VARIANT, MAX_VARIANTS,
paste(dropped_levels, collapse = ", "))
} else {
sprintf("Every group of '%s' had enough observations to keep. ", variant_h)
}
shape_label <- if (shape == "summary") "pre-aggregated totals" else "one row per visitor"
metrics <- list(
`Variants Compared` = as.integer(k),
`Total Observations` = as.integer(round(sum(trials))),
`Control Group` = control,
`Leading Variant` = leader,
`Leader Conversion Rate` = paste0(pctf(post_mean[lead_i]), "%"),
`P(Leader Is Best)` = fmt_prob(p_best_leader),
`P(Leader Beats Control)` = if (lead_i == ctrl_i) "n/a(the control leads)" else fmt_prob(p_beat[lead_i]),
`Expected Loss(Leader)` = paste0(ppf_min(e_loss[lead_i], 3), " pp"),
`Lift vs Control` = if (lead_i == ctrl_i) "n/a(the control leads)" else
paste0(ppf(lift[lead_i]), " pp(95% CrI ", ppf(lift_lo[lead_i]), " to ",
ppf(lift_hi[lead_i]), ")"),
`Decision Status` = if (decisive) "leader clears 0.95" else "not yet decided at 0.95",
`Data Shape` = shape_label
)
verdict <- if (decisive) {
sprintf("%s is the best arm with probability %s, and shipping it carries an expected loss of %s percentage points.",
leader, fmt_prob(p_best_leader), ppf_min(e_loss[lead_i], 3))
} else if (!separated) {
sprintf("No arm has separated: %s leads with P(best) = %s against %s's %s, a gap this data cannot resolve.",
leader, fmt_prob(p_best_leader), variants[runner_i], fmt_prob(p_best_runner))
} else {
sprintf("%s leads with P(best) = %s, short of the conventional 0.95 bar, and shipping it carries an expected loss of %s percentage points.",
leader, fmt_prob(p_best_leader), ppf_min(e_loss[lead_i], 3))
}
json_output <- list(
answer = paste0(
"Bayesian A/B test of ", k, " variants of '", variant_h, "' across ",
format(sum(trials), big.mark = ","), " observations(",
format(sum(conv), big.mark = ","), " conversions), read from ", shape_label,
". Posterior conversion rates run from ", pctf(min(post_mean)), "% to ",
pctf(max(post_mean)), "%; the control is ", control, ". ", verdict, " ",
if (focus_i != ctrl_i) paste0(
focus, " sits ", ppf(lift[focus_i]), " percentage points ",
if (lift[focus_i] >= 0) "above" else "below", " the control(95% credible interval ",
ppf(lift_lo[focus_i]), " to ", ppf(lift_hi[focus_i]), " percentage points), and beats it with probability ",
fmt_prob(p_beat[focus_i]), ". ") else "",
"These posteriors do not depend on how often you checked the test, but they ",
"also cannot correct for a stopping rule they were never told about; the ",
"prior contributes ", f3(prior_weight_pct), "% of the leader's posterior weight."
),
cards = lapply(
c("tldr", "overview", "preprocessing", "variant_posteriors",
"posterior_rates", "lift_distribution", "decision_table",
"priors_and_stopping", "methods"),
function(cid) list(id = cid, metrics = metrics)
)
)
list(
initial_rows = initial_rows, final_rows = final_rows, rows_removed = rows_removed,
variant_h = variant_h, outcome_h = outcome_h, trials_h = trials_h,
shape = shape, shape_label = shape_label, success_how = success_how,
variants = variants, trials = trials, conv = conv, k = k,
dropped_levels = dropped_levels, drop_note = drop_note,
n_missing_rows = n_missing_rows,
a0 = a0, b0 = b0, post_a = post_a, post_b = post_b,
post_mean = post_mean, ci_lo = ci_lo, ci_hi = ci_hi, obs_rate = obs_rate,
control = control, control_how = control_how, ctrl_i = ctrl_i,
p_beat = p_beat, p_best = p_best, e_loss = e_loss,
lift = lift, lift_lo = lift_lo, lift_hi = lift_hi,
rel_lift = rel_lift, rel_lo = rel_lo, rel_hi = rel_hi,
leader = leader, lead_i = lead_i, p_best_leader = p_best_leader,
runner_i = runner_i, p_best_runner = p_best_runner,
focus = focus, focus_i = focus_i,
decisive = decisive, separated = separated, mc_se_leader = mc_se_leader,
n_draws = N_DRAWS, pooled_rate = pooled_rate,
prior_weight_pct = prior_weight_pct, sens_df = sens_df,
sens_spread = sens_spread, sens_agree = sens_agree,
peek_line = peek_line, stop_rule_line = stop_rule_line,
prior_line = prior_line, threshold_line = threshold_line,
posteriors_df = posteriors_df, rates_df = rates_df,
lift_draws_df = lift_draws_df, decision_df = decision_df,
disclosure_df = disclosure_df,
verdict = verdict, metrics = metrics, json_output = json_output
)
}