What's the Probability B Is Actually Better? And What If You're Wrong?

How it works

Each variant's conversion rate is given a Beta posterior — Beta(prior + conversions, prior + non-conversions) — which is exact because the Beta prior is conjugate to the Binomial likelihood, so nothing is fitted and no normal approximation is used. The report draws 100,000 samples from each posterior with a fixed seed and reads the decision quantities straight off them: the share of draws where a variant exceeds the control, P(beats control); the share where it exceeds every other arm, P(best); and the average shortfall against the best arm on each draw, the expected loss. Credible intervals on the rates come from the Beta quantiles, and the interval on the lift comes from the draws. The prior is uniform Beta(1,1) by default, and the same conclusion is recomputed under a Jeffreys prior and a deliberately skeptical prior centred on the pooled rate so you can see how much of the answer is the prior rather than your data.

Use it when you want an experiment expressed as probabilities and costs rather than a significance verdict, when the test was monitored while running, or when you have only a handful of conversions per arm and a normal approximation would misbehave.

Not for a numeric outcome such as revenue per user or session length (this models a rate, not a mean) — use the A/B test tool for that. Not for repeated measures of the same person, and not as licence to stop a test the moment a number looks good: the posterior survives monitoring, a threshold you picked afterwards does not.

What data do you need?

One row per visitor: which variant they saw and whether they converted. A 0/1 flag, yes/no, or TRUE/FALSE all work:

Minimum 2 rows · Best with 1,000-200,000 rows across 2-4 variants, or 2-8 pre-aggregated summary rows

What's in the report?

Standard-library analysis: the Bayesian read on your experiment. Upload one row per visitor with the variant they saw and whether they converted, and get a Beta posterior for every variant's true conversion rate — the probability each variant beats the control, the probability each is the best of all, the expected loss (risk) of shipping each one, and the full posterior distribution of the lift rather than a single number. Because the posterior depends only on the totals, it does not go stale the way a fixed-horizon p-value does when you check the test while it runs — and the report says plainly what that does and does not protect you from, with the prior's influence measured against your own sample sizes. Already have per-variant totals? Map a trial-count column too and it reads pre-aggregated rows instead.