Is your A/B test result real?
Significance, lift and confidence interval

Statistical significance in A/B testing indicates whether the difference between your control and variant is larger than random variation alone would plausibly produce.

A test is typically considered statistically significant when the p-value is below 0.05 (95% confidence level), meaning that if there were no real difference, a result at least this extreme would occur less than 5% of the time.

MCP Analytics calculates this automatically using t-tests for continuous metrics (like revenue) and z-tests for proportions (like conversion rates), and provides plain-English interpretation so you don't need a statistics degree to understand the results.

You should stop an A/B test when you meet one of these criteria:

• Statistical significance reached: You've hit p < 0.05 AND collected your pre-determined minimum sample size
• Maximum duration reached: You've hit the maximum test duration you set beforehand (usually 2-4 weeks)
• Sequential analysis allows early stopping: Methods like Bayesian sequential testing indicate you can stop with high confidence

Warning: Never stop a test just because it looks like a winner early on. This practice, called "peeking," dramatically inflates false positive rates. If you check results 10 times during a test at p=0.05, your actual false positive rate is closer to 30%.

MCP Analytics provides guidance on when your test has sufficient data to make a reliable decision.

Sample size depends on four key factors:

• Baseline conversion rate: Your current metric (e.g., 5% conversion rate)
• Minimum detectable effect (MDE): The smallest improvement worth detecting (e.g., 10% relative lift)
• Statistical power: Typically 80% - the probability of detecting a real effect
• Significance level: Typically 95% (alpha = 0.05)

Example: To detect a 10% relative lift from a 5% baseline conversion rate with 80% power and 95% confidence, you need approximately 31,000 visitors per variant (62,000 total).

MCP Analytics includes a built-in sample size calculator that accounts for all these factors.

Frequentist A/B testing (traditional approach):

• Uses p-values and confidence intervals
• Answers: "Is there a statistically significant difference?"
• Requires fixed sample size determined in advance
• Can't be "peeked" at without inflating false positives

Bayesian A/B testing:

• Calculates the probability that one variant beats another
• Answers: "What's the probability B is better than A?" (e.g., 94%)
• Allows continuous monitoring without peeking problems
• Provides more intuitive results for business decisions

MCP Analytics provides both approaches so you can make the best decision for your situation. Bayesian is often preferred for its intuitive probability statements.

The standard threshold is p < 0.05, meaning that if there were no real difference, a result at least this extreme would occur less than 5% of the time (it is not a 95% chance that the result is real). The appropriate threshold depends on your situation:

• p < 0.10: Acceptable for low-risk changes or exploratory tests
• p < 0.05: Standard threshold for most business decisions
• p < 0.01: Recommended for high-stakes decisions like major redesigns or pricing changes

Important: Statistical significance alone doesn't guarantee practical significance. A test might show p=0.01 but only a 0.1% improvement - technically significant but not worth implementing. Always consider effect size and business impact alongside p-values.

A 95% confidence interval shows the range where the true effect likely falls. For example, if your A/B test shows a lift of +15% with a 95% CI of [+8%, +22%], the data are consistent with a true lift anywhere from 8% to 22%.

Key interpretations:

• If the CI doesn't include zero (or 1.0 for ratios), the result is statistically significant
• The width indicates precision - narrower intervals mean more reliable estimates
• For business decisions, focus on the lower bound as a conservative estimate

Example: If CI = [+2%, +25%], the data are consistent with a true lift of at least 2%. If CI = [-5%, +10%], the result is not significant because the interval includes zero (no effect).

Yes, but with caution. Running multiple tests on the same users creates interaction effects that can skew results. Best practices include:

• Mutually exclusive traffic: Allocate different user segments to different tests that might interact
• Multiple comparison corrections: Apply Bonferroni or FDR corrections when testing many variants
• Track experiment exposure: Know which users are in which experiments
• Consider multivariate testing: Use MVT for related changes to measure interactions

MCP Analytics supports multi-variant testing with up to 10 variants and automatically adjusts for multiple comparisons to maintain valid statistical conclusions.