Executive Summary
Key findings from the Welch independent-samples t-test comparing revenue between A/B variants
The difference in mean revenue between Variant A ($42.85) and Variant B ($48.06) is statistically significant (p = 0.000131). Cohen's d = -0.642, indicating a medium effect size. The 95% confidence interval for the mean difference is [-7.84, -2.59] USD.
Descriptive Statistics by Variant
Mean, standard deviation, sample size, and standard error of revenue per visitor for each experiment variant
| Variant | Revenue Usd |
|---|---|
| A | 42.85 |
| B | 48.06 |
Variant A had a mean revenue of $42.85 (SD = $8.37, n = 75, SE = $0.9666). Variant B had a mean revenue of $48.06 (SD = $7.88, n = 75, SE = $0.9102). The absolute mean difference between groups is $5.22.
T-Test Results and Effect Size
Welch t-statistic, degrees of freedom, p-value, confidence interval, and Cohen's d effect size
| Variant | Revenue Usd |
|---|---|
| T-Statistic | -3.9284 |
| Degrees of Freedom | 147.47 |
| P-Value | 0.0001309 |
| 95% CI (Mean Diff) | [-7.8397, -2.5921] |
| Cohen's d | -0.6415 |
| Effect Size | Medium |
| Result | Significant |
Welch's t-test yielded t(147.5) = -3.928, p = 0.000131. The 95% confidence interval for the mean difference runs from $-7.84 to $-2.59. Cohen's d = -0.642, representing a medium practical effect. The result is significant at α = 0.05.
Revenue Distribution by Variant
Box plots comparing the revenue distribution shape, median, and outliers between A/B variants
The box plot reveals the spread and central tendency of revenue for each variant. Variant A has a median revenue of $42.35 while Variant B has a median of $47.62. Outliers, if present, are shown as individual points beyond the whiskers and may influence the group means seen in the t-test.
Revenue Density Shape by Variant
Violin plots illustrating the distribution shape and density of revenue per visitor for each variant
Violin plots show the full probability density of revenue for each variant. A wider violin at a given revenue level indicates more visitors concentrated there. If the distributions appear roughly bell-shaped and symmetric, the normality assumption underlying the t-test is reasonably satisfied. Any skewness or multimodality should be noted as it may affect interpretation.
Mean Revenue by Variant
Bar chart comparing mean revenue per visitor between control and treatment variants
The bar chart compares mean revenue per visitor across the two experiment variants. Variant B leads by $5.22 per visitor. This difference is significant (p = 0.000131). The effect size (Cohen's d = -0.642) is classified as medium.
Revenue Distribution Overlay
Overlaid histograms of raw revenue values per visitor for each variant, supporting the normality assumption check
The overlaid histogram displays the raw revenue distribution for each variant, allowing visual inspection of normality and overlap. Welch's t-test is robust to moderate departures from normality when sample sizes are adequate. Substantial overlap between variants suggests a smaller practical effect, while well-separated distributions indicate a larger revenue difference.