The Verdict
Which of the 3 variants of 'Test Group' should you ship, and what does being wrong cost?
The short answer
Variant B is the best arm with probability greater than 0.999, converting at 13.99% versus Control's 9.59%—a lift of 4.40 percentage points. Shipping Variant B has an expected loss of 0.000 percentage points; staying on Control costs 4.400 percentage points on average.
The detail
Variant B beats Control with probability greater than 0.999, and the 95% credible interval for the lift (2.78 to 6.02 pp) sits entirely above zero. The whole credible interval is above zero, so the direction is settled; the size spans 3.25 percentage points, so a decision that flips within that range cannot be made from this data alone. The relative lift is 45.9% (95% credible interval 26.8% to 68.2%). These probabilities do not depend on how often you looked at the test while it ran, but they cannot correct for a stopping rule they were never told about. The prior contributes 0.067% of Variant B's posterior weight.
What this can't tell you
If you monitored the test and stopped the first time P(best) crossed the 0.95 bar, the reported probability greater than 0.999 overstates how often that decision rule would succeed in future tests. This file does not record when you looked, so the bias cannot be measured. Commit the stopping rule and minimum sample size before looking; do not reinterpret the posterior afterwards.
Analysis Overview
Beta-Binomial posterior comparison of 3 variants of 'Test Group' across 9,000 observations.
The short answer
This analysis uses a Bayesian model to estimate each variant's true conversion rate as a full probability distribution rather than a single point. Drawing 100,000 samples from each posterior distribution answers the business question directly: how likely is each variant to be the best, and how much conversion rate do you risk if you ship the wrong one?
The detail
The model is Beta-Binomial conjugate, so each posterior is exactly Beta(prior + conversions, prior + non-conversions). Across the 3 variants, posteriors range from Beta(288.00, 2582.00) to Beta(420.00, 2714.00). Each of the 100,000 posterior draws per variant supplies a plausible true rate; comparing draws across variants directly yields P(beats control), P(best), and expected loss. The Monte Carlo standard error on these probabilities is about 0.0000, and the seed is fixed at 42 so rerunning on the same file always gives the same answer.
What this can't tell you
The posterior depends only on the totals—1,009 conversions out of 9,000 observations—and is valid regardless of how many times you looked while the test ran. However, if you monitored and stopped the first time a threshold was crossed, the reported probabilities overstate how often that stopping rule would be right in future tests, and this single file cannot measure the bias. The fix is to commit to the rule and minimum sample size before looking, not to reinterpret the number afterwards.
Data Quality
How the file was read, what was dropped, and how conversions were counted.
The short answer
The file contains 9,078 rows read as one row per visitor. After dropping 78 rows with missing 'Test Group' or 'Converted' values and excluding Variant D (fewer than 5 observations), the analysis runs on 9,000 observations across 3 variants with 1,009 total conversions (pooled rate 11.21%).
The detail
Conversion was identified as a 1 in the 'Converted' column. Rows removed: 75 for missing values, 3 from Variant D (excluded as too small). Variants below the 5-observation threshold are dropped rather than pooled, because merging two different arms into one bucket would invent a variant nobody actually ran. Each row is assumed to be an independent visitor seeing one variant.
What this can't tell you
The file structure assumes each row is a single independent visit. If the same person visited multiple times and each visit is counted as a separate row, every confidence interval and credible interval below is narrower than it should be. Nothing in the file reveals repeat visitors, so this cannot be corrected from the data alone. Consider whether a visitor-level or session-level export would sharpen this assumption.
Posterior Rates by Variant
Observed and posterior conversion rate per variant, each with a 95% credible interval.
| Variant | Trials | Conversions | Observed Rate PCT | Posterior Rate PCT | CI Low PCT | CI High PCT |
|---|---|---|---|---|---|---|
| Control | 3000 | 287 | 9.567 | 9.594 | 8.566 | 10.67 |
| Variant B | 3000 | 419 | 13.97 | 13.99 | 12.77 | 15.25 |
| Variant C | 3000 | 303 | 10.1 | 10.13 | 9.073 | 11.23 |
The short answer
Variant B's posterior conversion rate is 13.99% (95% credible interval 12.77% to 15.25%), tightly pinned by 3,000 observations. Control is even tighter at 9.59% (8.57% to 10.67% on the same sample size), while Variant C sits at 10.13% (9.07% to 11.23%). The posterior rates sit slightly closer together than the raw rates because the prior shrinks every arm toward the middle.
The detail
Control: observed 9.567%, posterior 9.594%, interval width 2.11 pp (3,000 observations). Variant B: observed 13.967%, posterior 13.991%, interval width 2.48 pp (3,000 observations). Variant C: observed 10.1%, posterior 10.127%, interval width 2.16 pp (3,000 observations). The uniform Beta(1.00, 1.00) prior adds 2.00 pseudo-observations to each variant—0.067% of Variant B's total—enough to prevent a 1-for-2 arm from reading as 50%, but small enough to leave the data dominant.
What this can't tell you
Credible interval width reflects sample size; Variant B's is widest because all three arms have equal n. Tighter intervals do not imply better decisions—they measure precision, not accuracy. If the file contains repeat visitors coded as separate rows, the intervals are narrower than they should be.
Conversion Rate by Variant
Posterior conversion rate per variant with 95% credible intervals.
The short answer
Variant B leads at 13.99%, with Control at 9.59% and Variant C at 10.13%. The credible intervals show Variant B separated from both Control and Variant C, while Control and Variant C overlap. However, overlap is a rough visual guide; the draw-by-draw probabilities in the decision table are the real comparison.
The detail
Variant B's interval (12.77% to 15.25%) sits clearly above Control (8.57% to 10.67%) and Variant C (9.07% to 11.23%). Control and Variant C's intervals overlap. Interval width reflects sample size: all three variants have 3,000 observations, so whisker width is similar across arms. One of the 3 variant pairs have overlapping intervals, meaning some arms are separated on this data and some are not.
What this can't tell you
Visual overlap does not settle a comparison. Variant C's interval overlaps Control's, but the decision table shows P(beats control) = 0.756 for Variant C—the data has not decided that comparison. Conversely, no overlap does not guarantee separation; only the posterior draws reveal the full probability. Do not rely on whisker overlap to make a decision; use the expected-loss column instead.
Posterior Lift vs Control
The full posterior distribution of the difference in conversion rate, in percentage points.
The short answer
The posterior lift from Variant B over Control centers at 4.40 percentage points, with 95% of the mass between 2.78 and 6.02 pp. Greater than 0.999 of the posterior lies above zero, so the direction is certain; the size still is not—the interval spans 3.25 percentage points, so a decision that flips within that range is a decision this data cannot make.
The detail
The full posterior of the lift shows where the typical value sits (4.40 pp), how much of it is above zero (greater than 0.999), and the plausible range (2.78 to 6.02 pp, a spread of 3.25 pp). Because the whole interval is above zero, Variant B is better than Control in every posterior draw. The width of the interval tells you the size is not pinned down tightly: a decision rule that would flip somewhere inside the 2.78–6.02 range cannot be settled by this data alone.
What this can't tell you
The interval spans 3.25 percentage points. If your decision threshold lies within that range—for example, if you would ship Variant B only if the lift were at least 5 pp—this data cannot make that decision. A longer test, a higher conversion rate, or a larger effect would narrow the interval and firm up the answer. Consider whether the current interval width is acceptable for your business rule.
Decision Table
Probability of beating the control, probability of being best, and the expected loss of shipping each variant.
| Variant | Prob Beats Control | Prob Best | Expected Loss Pp | Lift Pp | Lift CI Low Pp | Lift CI High Pp | Decision Note |
|---|---|---|---|---|---|---|---|
| Control | — | 0 | 4.4 | 0 | — | — | The control. Shipping nothing costs an expected 4.400 percentage points of conversion rate against the best arm. |
| Variant B | 1 | 1 | 0 | 4.397 | 2.777 | 6.023 | Clears the conventional 0.95 bar for being best; the expected cost of shipping it is 0.000 percentage points. |
| Variant C | 0.7561 | 0 | 3.867 | 0.533 | -0.979 | 2.048 | Not separated from the control: P(beats control) = 0.756, so the data has not decided this comparison. |
The short answer
Variant B beats Control with probability 1.0000 and is best with probability 1.0000, carrying an expected loss of 0.000 percentage points. Variant C has P(beats control) = 0.7561, so the data has not separated it from Control. Staying on Control costs an expected 4.400 percentage points against the best arm.
The detail
Variant B: P(beats control) = 1.0000, P(best) = 1.0000, expected loss = 0.000 pp, lift = 4.397 pp (95% CrI 2.777 to 6.023 pp). Variant C: P(beats control) = 0.7561, P(best) = 0.0000, expected loss = 3.8669 pp, lift = 0.533 pp (95% CrI −0.979 to 2.048 pp). Control: expected loss = 4.3997 pp (the cost of shipping nothing when Variant B is best). The 0.95 bar for P(best) is a convention, not a law; expected loss is what a decision should actually turn on, because it is denominated in the thing you care about—conversion rate.
What this can't tell you
Variant C's P(beats control) of 0.7561 means the comparison is not settled; the interval includes zero (−0.979 to 2.048 pp), so the data does not rule out Control being better. Do not ship Variant C on this evidence. The expected-loss column assumes you will eventually ship one of these arms; if you would stay on Control anyway, compare its expected loss (4.400 pp) to the cost of running the test.
Priors, Peeking, and What This Cannot Tell You
The prior's influence, a sensitivity check across three priors, and an honest account of the stopping-rule problem.
| Prior | Leader Under Prior | Prob Leader Is Best | Leader Rate PCT | Expected Loss Pp |
|---|---|---|---|---|
| Uniform Beta(1.00, 1.00) — the prior used above | Variant B | 1 | 13.99 | 0 |
| Jeffreys Beta(0.50, 0.50) — the standard reference prior | Variant B | 1 | 13.98 | 0 |
| Skeptical Beta centred on the pooled rate 11.21%, worth 50 pseudo-trials per variant | Variant B | 1 | 13.92 | 0 |
The short answer
The prior Beta(1.00, 1.00) is uniform and contributes only 0.067% of Variant B's posterior weight. Under three different priors—uniform, Jeffreys, and a deliberately skeptical prior—Variant B remains the leader and P(best) does not change, so the conclusion is robust. However, if you monitored the test and stopped the first time P(best) crossed a bar, the reported probability greater than 0.999 overstates how often that rule would be right in future tests.
The detail
Prior sensitivity: uniform Beta(1.00, 1.00) yields P(best) = 1.0000 for Variant B at 13.993%; Jeffreys Beta(0.50, 0.50) yields P(best) = 1.0000 at 13.981%; skeptical Beta centered on 11.21% (50 pseudo-trials per variant) yields P(best) = 1.0000 at 13.923%. Across all three priors, the leader does not change and P(best) moves by 0.000. The posterior depends only on totals—1,009 conversions out of 9,000—not on how many times you looked. This is a real advantage over fixed-horizon p-values.
What this can't tell you
This file was run once and does not record when you looked. If you monitored and stopped the first time P(best) crossed 0.95, the reported greater than 0.999 overstates the long-run success rate of that stopping rule, and nothing in this data can measure the bias. The fix is to commit the rule and minimum sample size before you look, not to reinterpret the number afterwards. Whether an expected loss of 0.000 pp is small enough to act on is a business judgement; the analysis computes the risk but cannot set the threshold.
Methods & Disclosure
The model, the prior, the draws, the control, and the limits of what this analysis can decide.
| Item | Detail |
|---|---|
| Model | Beta-Binomial conjugate: each variant's conversion rate gets a Beta posterior with parameters (prior + conversions, prior + non-conversions). No model is fitted and no normal approximation is used, so the arms with few observations are handled correctly. |
| Prior | Beta(1.00, 1.00), uniform over the conversion rate. Posterior parameters run from Beta(288.00, 2582.00) to Beta(420.00, 2714.00) across the 3 variants. |
| Posterior draws | 100,000 draws per variant from each posterior, seed 42, so re-running this analysis on the same file returns the same probabilities. |
| Monte Carlo precision | The Monte Carlo standard error on Variant B's P(best) of 1.0000 is 0.0000, so the third decimal place is noise from the simulation rather than signal from the data. |
| Control group | 'Control' (its name identifies it as the control). Every lift, credible interval and P(beats control) in this report is measured against it. |
| Data shape | One row per visitor or session, aggregated to per-variant totals by this analysis. 'Converted' holds only 0 and 1; 1 was counted as a conversion. |
| How conversions were counted | 1,009 conversions out of 9,000 observations overall, a pooled rate of 11.21%. |
| Does monitoring invalidate this? | The posterior depends only on the totals — 1,009 conversions out of 9,000 observations — and not on how many times you looked while the test ran. That is a real advantage over a fixed-horizon p-value, which is only valid for the sample size you committed to in advance. It is NOT a licence to stop whenever the number looks good. |
| What this cannot correct for | This analysis was run once, on a file that does not record when you looked. If you monitored the test and stopped the first time P(best) crossed a bar, the reported greater than 0.999 for Variant B overstates how often that decision rule would be right, and nothing in this data can measure by how much. The fix is to fix the rule — the bar, and the minimum sample — before you look, not to reinterpret the number afterwards. |
| Who sets the threshold | The expected loss of shipping Variant B is 0.000 percentage points of conversion rate. Whether that is small enough to act on is a business judgement about what a percentage point is worth to you — the analysis computes the risk, it cannot set the threshold. |
The short answer
The data yields a posterior probability greater than 0.999 that Variant B outperforms Control. The expected loss from shipping Variant B is 0.000 percentage points of conversion rate — meaning the analysis detects no material risk to moving forward. This verdict rests on a Beta-Binomial conjugate model with a uniform prior, 100,000 posterior draws, and 1,009 conversions across 9,000 total observations.
The detail
Each variant's conversion rate posterior is Beta(prior + conversions, prior + non-conversions), with Variant B at Beta(420.00, 2582.00). The prior is Beta(1.00, 1.00), uniform and carrying negligible weight relative to the observed data. All probabilities are the exact share of 100,000 posterior draws satisfying the stated condition, seeded at 42 for reproducibility; the Monte Carlo standard error on Variant B's P(best) of 1.0000 is 0.0000. The control group is identified by name ('Control'). Conversions were counted as 1 in a binary field; the pooled rate was 11.21%.
What this can't tell you
If the test was monitored and stopped when P(best) crossed a threshold, the reported probability greater than 0.999 overstates the long-run accuracy of that decision rule — the posterior depends only on totals, not stopping time, so sequential peeking is not automatically corrected. Setting the threshold at which expected loss of 0.000 percentage points is acceptable is a business decision outside the analysis. Consider pre-specifying the decision rule and minimum sample size before interpreting results from future tests.
Bayesian A/B Test — Beta-Binomial Conjugate Analysis
Answers the question a frequentist significance test cannot: given what you have seen, what is the probability each variant is the best one, and what does it cost you to be wrong? Each variant's conversion rate gets a Beta posterior (conjugate to the Binomial likelihood); posterior draws then give P(variant beats control), P(variant is best), the expected loss (risk) of shipping each variant, and credible intervals on the lift.
Why This Method?
A p-value answers "how surprising would this data be if nothing were happening?" — a question nobody asked. The posterior answers the question people actually ask: how likely is B better than A, by how much, and what do I lose if I pick wrong. Because the Beta prior is conjugate to the Binomial likelihood, the posterior has a closed form and needs no fitting.
What This Analysis Covers
- Posterior conversion rate per variant with a 95% credible interval
- P(variant beats control) and P(variant is the best of all)
- Expected loss (risk) of shipping each variant
- The full posterior distribution of the lift, not just a point estimate
- An honest treatment of the "no peeking problem": what the posterior
really is and is not immune to, with the prior's influence quantified from the user's own sample sizes
Standard Library
Platform standard-library module (LAT-1441): runs on ANY dataset via the semantic mapping {variant, outcome, trials}. All narrative is derived from the user's own column names and computed values.
suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))Core Analysis Pipeline
compute_shared <- function(df, params, col_map = list()) {
# === SHARED EXPORTS ===
# initial_rows/final_rows/rows_removed $ row accounting
# variant_h / outcome_h / trials_h $ humanized user column names
# shape / shape_label $ "row-per-unit" | "summary" + prose label
# success_how $ how the success level was determined (prose)
# variants $ character — kept variant labels (control first)
# trials / conv $ integer — per kept variant
# dropped_levels / drop_note $ variants excluded + why
# n_missing_rows $ rows dropped for a missing variant/outcome
# a0 / b0 $ the prior's Beta parameters
# post_a / post_b $ posterior Beta parameters per variant
# post_mean / ci_lo / ci_hi $ posterior rate + 95% credible interval
# control / control_how $ control label + how it was chosen
# p_beat / p_best / e_loss $ decision quantities per variant
# lift / lift_lo / lift_hi $ absolute lift vs control (proportion units)
# rel_lift / rel_lo / rel_hi $ relative lift vs control (fraction)
# leader / focus $ highest P(best) / best non-control variant
# decisive / separated $ leader clears 0.95 / clears the runner-up
# mc_se_leader $ Monte Carlo standard error on the leader's P(best)
# prior_weight_pct $ % of the leader's posterior weight from the prior
# sens_df $ P(best) for the leader under 3 priors
# posteriors_df / rates_df / lift_draws_df / decision_df / disclosure_df
# metrics / json_output
# === /SHARED EXPORTS ===
N_DRAWS <- 100000L # posterior draws per variant (seed fixed below)
MAX_VARIANTS <- 8L # keep at most this many arms
MIN_TRIALS_PER_VARIANT <- 5L
MIN_TOTAL_TRIALS <- 20L
SKEPTICAL_WEIGHT <- 50 # pseudo-trials per variant in the skeptical prior
variant_h <- humanize_semantic("variant", col_map)
outcome_h <- humanize_semantic("outcome", col_map)
trials_h <- humanize_semantic("trials", col_map)Step 1: Check the mapping
initial_rows <- nrow(df)
if (!("variant" %in% names(df))) {
stop(sprintf("This analysis needs a variant column mapped — the column that says which version each row belongs to(mapped as '%s').",
variant_h))
}
if (!("outcome" %in% names(df))) {
stop(sprintf("This analysis needs an outcome column mapped — the conversion flag, or the conversion COUNT if your rows are already totalled by variant(mapped as '%s').",
outcome_h))
}Step 2: Which shape is this — one row per unit, or pre-aggregated totals?
The widget maps two columns (variant + outcome) and treats every row as one visitor. Mapping a third column of trial counts switches the module to summary mode, where each row is a whole variant.
shape <- if ("trials" %in% names(df)) "summary" else "row-per-unit"
v_all <- trimws(as.character(df$variant))
v_ok <- !is.na(v_all) & v_all != ""
success_how <- ""
if (shape == "summary") {Summary shape: outcome = conversions, trials = trials, one row per variant.
coerce_count <- function(v, label_h) {
ch <- trimws(as.character(v))
non_blank <- !is.na(ch) & ch != ""
conv <- suppressWarnings(as.numeric(ch))
if (sum(non_blank) == 0 ||
sum(!is.na(conv[non_blank])) < 0.95 * sum(non_blank)) {
stop(sprintf("The column '%s' does not look numeric — fewer than 95%% of its values parse as numbers. In summary mode it must hold whole counts.",
label_h))
}
conv
}
tr_all <- coerce_count(df$trials, trials_h)
cv_all <- coerce_count(df$outcome, outcome_h)
ok <- v_ok & !is.na(tr_all) & !is.na(cv_all)
n_missing_rows <- sum(!ok)
vv <- v_all[ok]; tt <- tr_all[ok]; cc <- cv_all[ok]
if (length(vv) == 0) {
stop(sprintf("No rows have a usable '%s', '%s' and '%s' together.",
variant_h, outcome_h, trials_h))
}
if (any(tt < 0) || any(cc < 0)) {
stop(sprintf("'%s' and '%s' must be counts of zero or more — negative values were found.",
outcome_h, trials_h))
}
bad <- which(cc > tt)
if (length(bad) > 0) {
stop(sprintf("'%s' exceeds '%s' for %d row(s) (for example %s: %s conversions out of %s). Conversions cannot outnumber the people who saw the variant.",
outcome_h, trials_h, length(bad), vv[bad[1]],
format(cc[bad[1]]), format(tt[bad[1]])))
}
agg_tr <- tapply(tt, vv, sum)
agg_cv <- tapply(cc, vv, sum)
success_how <- sprintf("Each row of your file is a whole variant: '%s' is the number of people who saw it and '%s' is the number who converted.",
trials_h, outcome_h)
} else {Row-per-unit shape: every row is one visitor; the outcome is a flag.
o_all <- df$outcome
o_is_logical <- is.logical(o_all)
o_chr_all <- trimws(as.character(o_all))
ok <- v_ok & !is.na(o_all) & !is.na(o_chr_all) & o_chr_all != ""
n_missing_rows <- sum(!ok)
vv <- v_all[ok]
o_chr <- o_chr_all[ok]
if (length(vv) == 0) {
stop(sprintf("No rows have both a '%s' and an '%s' value.", variant_h, outcome_h))
}
y <- NULL
if (o_is_logical) {
y <- as.numeric(o_all[ok])
success_how <- sprintf("'%s' is a TRUE/FALSE column; TRUE was counted as a conversion.", outcome_h)
} else {
num <- suppressWarnings(as.numeric(o_chr))
if (sum(!is.na(num)) >= 0.95 * length(o_chr)) {
keep_num <- !is.na(num)
vv <- vv[keep_num]; num <- num[keep_num]
n_missing_rows <- n_missing_rows + sum(!keep_num)
uu <- sort(unique(num))
if (!all(uu %in% c(0, 1))) {
stop(sprintf("'%s' holds %d distinct numeric values (for example %s and %s). A Bayesian A/B test models a CONVERSION RATE, so it needs a 0/1 flag per row — or map a trial-count column as well and give it conversion counts per variant.",
outcome_h, length(uu), f2(uu[1]), f2(uu[length(uu)])))
}
y <- num
success_how <- sprintf("'%s' holds only 0 and 1; 1 was counted as a conversion.", outcome_h)
} else {
lv <- sort(unique(o_chr))
if (length(lv) != 2) {
stop(sprintf("'%s' has %d distinct non-numeric values, so it cannot be read as a conversion flag. Map a 0/1 column, a two-level yes/no column, or add a trial-count column and supply conversion counts.",
outcome_h, length(lv)))
}
kw <- c("yes", "y", "true", "t", "converted", "conversion", "success",
"won", "win", "clicked", "purchased", "paid", "subscribed",
"signup", "signed up", "1")
hit <- lv[tolower(lv) %in% kw]
if (length(hit) >= 1) {
success_label <- hit[1]
success_how <- sprintf("'%s' has two values ('%s' and '%s'); '%s' was counted as a conversion because it matches a known success word.",
outcome_h, lv[1], lv[2], success_label)
} else {
success_label <- lv[2]
success_how <- sprintf("'%s' has two values ('%s' and '%s') and neither matches a known success word, so '%s' — the alphabetically later one — was counted as the conversion. Check that this is the direction you meant.",
outcome_h, lv[1], lv[2], success_label)
}
y <- as.numeric(o_chr == success_label)
}
}
agg_tr <- tapply(rep(1, length(y)), vv, sum)
agg_cv <- tapply(y, vv, sum)
}Step 3: Minimum-size guards, named in the user's own columns
all_levels <- names(agg_tr)
trials_all <- as.numeric(agg_tr[all_levels])
conv_all <- as.numeric(agg_cv[all_levels])
total_trials_all <- sum(trials_all)
if (total_trials_all < MIN_TOTAL_TRIALS) {
stop(sprintf("Only %s observation(s) of '%s' are usable — a Bayesian A/B test needs at least %d in total, and at least %d per group of '%s'.",
format(total_trials_all, big.mark = ","), variant_h,
MIN_TOTAL_TRIALS, MIN_TRIALS_PER_VARIANT, variant_h))
}Keep the largest arms; drop tiny ones rather than pooling them (pooling distinct variants into an "Other" bucket would invent a variant nobody ran).
ord <- order(-trials_all, all_levels)
ranked <- all_levels[ord]
beyond_cap <- if (length(ranked) > MAX_VARIANTS) ranked[(MAX_VARIANTS + 1):length(ranked)] else character(0)
candidates <- setdiff(ranked, beyond_cap)
too_small <- candidates[as.numeric(agg_tr[candidates]) < MIN_TRIALS_PER_VARIANT]
variants <- setdiff(candidates, too_small)
dropped_levels <- c(beyond_cap, too_small)
if (length(variants) < 2) {
stop(sprintf("After cleaning, '%s' has %d group(s) with at least %d observation(s) — a comparison needs at least two.",
variant_h, length(variants), MIN_TRIALS_PER_VARIANT))
}
trials <- as.numeric(agg_tr[variants])
conv <- as.numeric(agg_cv[variants])
if (sum(conv) == 0) {
stop(sprintf("Not one of the %s observations converted — every value of '%s' is a non-conversion, so there is no conversion rate to compare.",
format(sum(trials), big.mark = ","), outcome_h))
}
final_rows <- if (shape == "summary") length(variants) else sum(trials)
rows_removed <- initial_rows - final_rowsStep 4: Choose the control — a group named like one, else the largest
named_ctrl <- variants[tolower(variants) %in%
c("control", "a", "baseline", "original", "ctrl", "holdout")]
control <- if (length(named_ctrl) > 0) named_ctrl[1] else variants[safe_which_max(trials)]
control_how <- if (length(named_ctrl) > 0) {
sprintf("'%s' (its name identifies it as the control)", control)
} else {
sprintf("'%s' (the largest group, since no group is named like a control)", control)
}Put the control first so every table reads control-then-challengers.
variants <- c(control, setdiff(variants, control))
trials <- as.numeric(agg_tr[variants])
conv <- as.numeric(agg_cv[variants])
k <- length(variants)
ctrl_i <- 1LStep 5: Conjugate posteriors — Beta(a0 + conversions, b0 + failures)
a0 <- 1; b0 <- 1 # uniform prior over the conversion rate
post_a <- a0 + conv
post_b <- b0 + (trials - conv)
post_mean <- post_a / (post_a + post_b)
ci_lo <- stats::qbeta(0.025, post_a, post_b)
ci_hi <- stats::qbeta(0.975, post_a, post_b)
obs_rate <- conv / trialsStep 6: Posterior draws — every decision quantity comes from these
set.seed(42)
draw_matrix <- function(aa, bb) {
matrix(stats::rbeta(N_DRAWS * length(aa),
rep(aa, each = N_DRAWS),
rep(bb, each = N_DRAWS)),
nrow = N_DRAWS, ncol = length(aa))
}
D <- draw_matrix(post_a, post_b)
best_idx <- max.col(D, ties.method = "first")
row_max <- D[cbind(seq_len(N_DRAWS), best_idx)]
p_best <- as.numeric(tabulate(best_idx, nbins = k)) / N_DRAWS
e_loss <- colMeans(row_max - D)
ctrl_draws <- D[, ctrl_i]
p_beat <- colMeans(D > ctrl_draws)
p_beat[ctrl_i] <- NA_real_
lift <- post_mean - post_mean[ctrl_i] # exact posterior mean of the difference
lift_lo <- rep(NA_real_, k); lift_hi <- rep(NA_real_, k)
rel_lift <- rep(NA_real_, k); rel_lo <- rep(NA_real_, k); rel_hi <- rep(NA_real_, k)
for (i in seq_len(k)) {
if (i == ctrl_i) next
dl <- D[, i] - ctrl_draws
q <- stats::quantile(dl, c(0.025, 0.975), names = FALSE)
lift_lo[i] <- q[1]; lift_hi[i] <- q[2]
rl <- dl / ctrl_draws
qr <- stats::quantile(rl, c(0.025, 0.5, 0.975), names = FALSE)
rel_lo[i] <- qr[1]; rel_lift[i] <- qr[2]; rel_hi[i] <- qr[3]
}Step 7: Who leads, and is the lead real enough to act on?
lead_i <- safe_which_max(p_best)
leader <- variants[lead_i]
p_best_leader <- p_best[lead_i]
runner_i <- safe_which_max(replace(p_best, lead_i, NA_real_))
p_best_runner <- p_best[runner_i]
mc_se_leader <- sqrt(p_best_leader * (1 - p_best_leader) / N_DRAWS)
decisive <- p_best_leader >= 0.95
separated <- (p_best_leader - p_best_runner) > 0.05The reported lift picture always compares a challenger with the control.
non_ctrl <- setdiff(seq_len(k), ctrl_i)
focus_i <- non_ctrl[safe_which_max(p_best[non_ctrl])]
focus <- variants[focus_i]Step 8: How much of the answer is the prior, not the data?
prior_weight_pct <- 100 * (a0 + b0) / (a0 + b0 + trials[lead_i])
pooled_rate <- sum(conv) / sum(trials)
prior_defs <- list(
list(label = sprintf("Uniform Beta(%s, %s) — the prior used above", f2(a0), f2(b0)),
a = rep(a0, k), b = rep(b0, k)),
list(label = "Jeffreys Beta(0.50, 0.50) — the standard reference prior",
a = rep(0.5, k), b = rep(0.5, k)),
list(label = sprintf("Skeptical Beta centred on the pooled rate %s%%, worth %s pseudo-trials per variant",
pctf(pooled_rate), format(SKEPTICAL_WEIGHT)),
a = rep(SKEPTICAL_WEIGHT * pooled_rate, k),
b = rep(SKEPTICAL_WEIGHT * (1 - pooled_rate), k))
)
sens_rows <- list()
for (pd in prior_defs) {
set.seed(42)
Ds <- draw_matrix(pd$a + conv, pd$b + (trials - conv))
bi <- max.col(Ds, ties.method = "first")
pb <- as.numeric(tabulate(bi, nbins = k)) / N_DRAWS
rm_s <- Ds[cbind(seq_len(N_DRAWS), bi)]
el <- colMeans(rm_s - Ds)
sens_rows[[length(sens_rows) + 1]] <- data.frame(
prior = pd$label,
leader_under_prior = variants[safe_which_max(pb)],
prob_leader_is_best = round(max(pb), 4),
leader_rate_pct = round(100 * mean(Ds[, safe_which_max(pb)]), 3),
expected_loss_pp = round(100 * el[safe_which_max(pb)], 4),
stringsAsFactors = FALSE
)
}
sens_df <- do.call(rbind, sens_rows)
rownames(sens_df) <- NULL
sens_spread <- max(sens_df$prob_leader_is_best) - min(sens_df$prob_leader_is_best)
sens_agree <- length(unique(sens_df$leader_under_prior)) == 1Step 10: Disclosure — what the posterior is and is not immune to
peek_line <- paste0(
"The posterior depends only on the totals — ", format(sum(conv), big.mark = ","),
" conversions out of ", format(sum(trials), big.mark = ","),
" observations — and not on how many times you looked while the test ran. ",
"That is a real advantage over a fixed-horizon p-value, which is only valid ",
"for the sample size you committed to in advance. It is NOT a licence to stop ",
"whenever the number looks good."
)
stop_rule_line <- paste0(
"This analysis was run once, on a file that does not record when you looked. ",
"If you monitored the test and stopped the first time P(best) crossed a bar, ",
"the reported ", fmt_prob(p_best_leader), " for ", leader,
" overstates how often that decision rule would be right, and nothing in this ",
"data can measure by how much. The fix is to fix the rule — the bar, and the ",
"minimum sample — before you look, not to reinterpret the number afterwards."
)
prior_line <- paste0(
"The prior adds ", f2(a0 + b0), " pseudo-observations to each variant, which is ",
f3(prior_weight_pct), "% of ", leader, "'s ", format(trials[lead_i], big.mark = ","),
" observations. Under the three priors tested, P(best) for the leading variant spans ",
f3_min(sens_spread), " and the leading variant ",
if (sens_agree) "does not change" else "CHANGES, so the conclusion is prior-driven, not data-driven",
"."
)
threshold_line <- paste0(
"The expected loss of shipping ", leader, " is ", ppf_min(e_loss[lead_i], 3),
" percentage points of conversion rate. Whether that is small enough to act on ",
"is a business judgement about what a percentage point is worth to you — the ",
"analysis computes the risk, it cannot set the threshold."
)
disclosure_df <- data.frame(
item = c("Model", "Prior", "Posterior draws", "Monte Carlo precision",
"Control group", "Data shape", "How conversions were counted",
"Does monitoring invalidate this?", "What this cannot correct for",
"Who sets the threshold"),
detail = c(
sprintf("Beta-Binomial conjugate: each variant's conversion rate gets a Beta posterior with parameters (prior + conversions, prior + non-conversions). No model is fitted and no normal approximation is used, so the arms with few observations are handled correctly.")
,
sprintf("Beta(%s, %s), uniform over the conversion rate. Posterior parameters run from Beta(%s, %s) to Beta(%s, %s) across the %d variants.",
f2(a0), f2(b0), f2(min(post_a)), f2(min(post_b)),
f2(max(post_a)), f2(max(post_b)), k),
sprintf("%s draws per variant from each posterior, seed 42, so re-running this analysis on the same file returns the same probabilities.",
format(N_DRAWS, big.mark = ",")),
sprintf("The Monte Carlo standard error on %s's P(best) of %s is %s, so the third decimal place is noise from the simulation rather than signal from the data.",
leader, f4(p_best_leader), f4(mc_se_leader)),
sprintf("%s. Every lift, credible interval and P(beats control) in this report is measured against it.", control_how),
sprintf("%s %s", if (shape == "summary")
"Pre-aggregated totals: one row per variant." else
"One row per visitor or session, aggregated to per-variant totals by this analysis.",
success_how),
sprintf("%s conversions out of %s observations overall, a pooled rate of %s%%.",
format(sum(conv), big.mark = ","), format(sum(trials), big.mark = ","),
pctf(pooled_rate)),
peek_line,
stop_rule_line,
threshold_line
),
stringsAsFactors = FALSE
)
drop_note <- if (length(dropped_levels) > 0) {
sprintf("%d group(s) of '%s' were excluded (fewer than %d observations, or beyond the %d largest arms): %s. ",
length(dropped_levels), variant_h, MIN_TRIALS_PER_VARIANT, MAX_VARIANTS,
paste(dropped_levels, collapse = ", "))
} else {
sprintf("Every group of '%s' had enough observations to keep. ", variant_h)
}
shape_label <- if (shape == "summary") "pre-aggregated totals" else "one row per visitor"
metrics <- list(
`Variants Compared` = as.integer(k),
`Total Observations` = as.integer(round(sum(trials))),
`Control Group` = control,
`Leading Variant` = leader,
`Leader Conversion Rate` = paste0(pctf(post_mean[lead_i]), "%"),
`P(Leader Is Best)` = fmt_prob(p_best_leader),
`P(Leader Beats Control)` = if (lead_i == ctrl_i) "n/a(the control leads)" else fmt_prob(p_beat[lead_i]),
`Expected Loss(Leader)` = paste0(ppf_min(e_loss[lead_i], 3), " pp"),
`Lift vs Control` = if (lead_i == ctrl_i) "n/a(the control leads)" else
paste0(ppf(lift[lead_i]), " pp(95% CrI ", ppf(lift_lo[lead_i]), " to ",
ppf(lift_hi[lead_i]), ")"),
`Decision Status` = if (decisive) "leader clears 0.95" else "not yet decided at 0.95",
`Data Shape` = shape_label
)
verdict <- if (decisive) {
sprintf("%s is the best arm with probability %s, and shipping it carries an expected loss of %s percentage points.",
leader, fmt_prob(p_best_leader), ppf_min(e_loss[lead_i], 3))
} else if (!separated) {
sprintf("No arm has separated: %s leads with P(best) = %s against %s's %s, a gap this data cannot resolve.",
leader, fmt_prob(p_best_leader), variants[runner_i], fmt_prob(p_best_runner))
} else {
sprintf("%s leads with P(best) = %s, short of the conventional 0.95 bar, and shipping it carries an expected loss of %s percentage points.",
leader, fmt_prob(p_best_leader), ppf_min(e_loss[lead_i], 3))
}
json_output <- list(
answer = paste0(
"Bayesian A/B test of ", k, " variants of '", variant_h, "' across ",
format(sum(trials), big.mark = ","), " observations(",
format(sum(conv), big.mark = ","), " conversions), read from ", shape_label,
". Posterior conversion rates run from ", pctf(min(post_mean)), "% to ",
pctf(max(post_mean)), "%; the control is ", control, ". ", verdict, " ",
if (focus_i != ctrl_i) paste0(
focus, " sits ", ppf(lift[focus_i]), " percentage points ",
if (lift[focus_i] >= 0) "above" else "below", " the control(95% credible interval ",
ppf(lift_lo[focus_i]), " to ", ppf(lift_hi[focus_i]), " percentage points), and beats it with probability ",
fmt_prob(p_beat[focus_i]), ". ") else "",
"These posteriors do not depend on how often you checked the test, but they ",
"also cannot correct for a stopping rule they were never told about; the ",
"prior contributes ", f3(prior_weight_pct), "% of the leader's posterior weight."
),
cards = lapply(
c("tldr", "overview", "preprocessing", "variant_posteriors",
"posterior_rates", "lift_distribution", "decision_table",
"priors_and_stopping", "methods"),
function(cid) list(id = cid, metrics = metrics)
)
)
list(
initial_rows = initial_rows, final_rows = final_rows, rows_removed = rows_removed,
variant_h = variant_h, outcome_h = outcome_h, trials_h = trials_h,
shape = shape, shape_label = shape_label, success_how = success_how,
variants = variants, trials = trials, conv = conv, k = k,
dropped_levels = dropped_levels, drop_note = drop_note,
n_missing_rows = n_missing_rows,
a0 = a0, b0 = b0, post_a = post_a, post_b = post_b,
post_mean = post_mean, ci_lo = ci_lo, ci_hi = ci_hi, obs_rate = obs_rate,
control = control, control_how = control_how, ctrl_i = ctrl_i,
p_beat = p_beat, p_best = p_best, e_loss = e_loss,
lift = lift, lift_lo = lift_lo, lift_hi = lift_hi,
rel_lift = rel_lift, rel_lo = rel_lo, rel_hi = rel_hi,
leader = leader, lead_i = lead_i, p_best_leader = p_best_leader,
runner_i = runner_i, p_best_runner = p_best_runner,
focus = focus, focus_i = focus_i,
decisive = decisive, separated = separated, mc_se_leader = mc_se_leader,
n_draws = N_DRAWS, pooled_rate = pooled_rate,
prior_weight_pct = prior_weight_pct, sens_df = sens_df,
sens_spread = sens_spread, sens_agree = sens_agree,
peek_line = peek_line, stop_rule_line = stop_rule_line,
prior_line = prior_line, threshold_line = threshold_line,
posteriors_df = posteriors_df, rates_df = rates_df,
lift_draws_df = lift_draws_df, decision_df = decision_df,
disclosure_df = disclosure_df,
verdict = verdict, metrics = metrics, json_output = json_output
)
}Your turn
Bring your own data and the question you actually need answered.
CympleData Scientist Send me your data and question, I’ll send you the analytics. ds@mcpanalytics.ai