Executive Summary
Key findings from the A/B test experiment
This A/B test analyzed 1,200 sessions across 3 variants. The best-performing variant 'variant_a' achieved a 4.13% conversion rate versus 4.12% for control (a 0.01 percentage-point lift). No variants reached statistical significance. At NULL annual sessions, the maximum projected revenue lift is $0.
Conversion Rate by Variant
Conversion rate, total sessions, and conversions per experiment variant
Conversion rates are shown for all 3 variants. The highest conversion rate is 4.13% for variant 'variant_a', compared to 4.12% for control. The absolute lift is 0.01 percentage points. Session counts allow you to assess whether differences are driven by sample-size imbalance.
Statistical Significance Test
Chi-square p-values, test statistics, and odds ratios for each variant vs control
| comparison | p_value | chi_square | odds_ratio | significant | _row |
|---|---|---|---|---|---|
| variant_a vs control | 0.9897 | 0.0002 | 1.005 | No | variant_a |
| variant_b vs control | 0.1305 | 2.286 | 0.5362 | No | variant_b |
Chi-square tests compare each variant's conversion rate to control at the 95% confidence level (alpha = 0.05). 0 out of 2 variant comparison(s) reached statistical significance. The odds ratio quantifies how much more likely a visitor in the variant group is to convert versus control. An odds ratio above 1 indicates the variant outperforms control.
Lift Confidence Intervals
95% confidence intervals for conversion rate lift vs control
Each point shows the estimated percentage-point lift of that variant over control, with 95% confidence intervals from the normal approximation. 0 out of 2 variant(s) have confidence intervals entirely above zero, indicating a reliably positive effect. Intervals crossing zero indicate insufficient evidence to conclude the variant improves conversion.
Projected Annual Revenue Impact
Annual revenue projection at 1e+05 sessions per year
Revenue projections assume the observed average revenue per session scales to 1e+05 annual sessions. The highest projected annual revenue is $15,207,416 for variant 'variant_a'. This represents a $587,499 decrease versus control. Projections capture both conversion rate differences and average order value per session.
Average Revenue per Session by Variant
Revenue efficiency per session and per converting session, by variant
Average revenue per session captures the combined effect of conversion rate and average order value. Variant 'variant_a' generates $152.07 per session on average, compared to $157.95 for control. The 'avg revenue per conversion' label shows revenue among converting sessions only, revealing whether order value — not just conversion rate — differs across variants.
Sample Size & Power Analysis
Observed statistical power and required sample sizes for each variant vs control
| variant | actual_n | observed_power | adequately_powered | required_n_per_group | _row |
|---|---|---|---|---|---|
| variant_a | 387 | 0.0258 | No | 18859174 | variant_a |
| variant_b | 400 | 0.3303 | No | 1386 | variant_b |
Power analysis evaluates whether each variant comparison is adequately powered (>=80% power) to detect the observed conversion rate difference. 0 out of 2 comparison(s) are adequately powered. The 'required n per group' column shows the minimum sample size per variant to achieve 80% power. Under-powered experiments risk missing real effects (Type II error).
Engagement: Pages Viewed by Variant & Conversion
Average pages viewed by variant, split between converting and non-converting sessions
Pages viewed serves as a proxy for session engagement depth. On average, converting sessions view 7.4 pages versus 8.1 pages for non-converting sessions (ratio: 0.91x). Differences in this ratio across variants may indicate whether certain variants attract users who browse more before converting, or accelerate conversion with fewer page views.