Power Analysis: How Big a Sample Do You Need?

Consider a made-up team that ran a test and found no clear gap, then saw a similar feature gain attention elsewhere. A result with no clear gap alone does not show that the feature had no effect. The test may have had too little power to find an effect that mattered.

Power planning can help teams plan which effects their tests could find. Its value depends on the inputs and assumptions.

Worked Example (Simulated)

Question: At a 2.5% buy rate, how often would a two-sided test find a drop in the buy rate after a price rise?

A team tests a 10% price rise. Finance wants revenue to rise at least 3%. Revenue ratio = price ratio x volume ratio, so 1.03 = 1.10 x volume ratio, and the volume ratio is 0.936. That is a 6.4% drop in the buy rate at most, so 6.4% is the drop to find; 10% is a larger drop for comparison. The script draws simulated visitors who either buy or do not, in equal control and test groups. A run counts as a win when the pooled z test passes its two-sided cut-off. It needs numpy and scipy.

import numpy as np
from scipy.stats import norm

seed = 20261029
replicates = 100_000
p_control = 0.025
alpha = 0.05
z_critical = norm.ppf(1 - alpha / 2)
sample_sizes = (10_000, 20_000, 40_000, 60_000)
relative_drops = (0.064, 0.10)
rng = np.random.default_rng(seed)


def simulated_power(n_per_arm, relative_drop):
    p_treatment = p_control * (1 - relative_drop)
    control_success = rng.binomial(n_per_arm, p_control, size=replicates)
    treatment_success = rng.binomial(n_per_arm, p_treatment, size=replicates)
    pooled = (control_success + treatment_success) / (2 * n_per_arm)
    standard_error = np.sqrt(pooled * (1 - pooled) * (2 / n_per_arm))
    diff = treatment_success / n_per_arm - control_success / n_per_arm
    z = diff / standard_error
    return np.mean(np.abs(z) > z_critical)


print(f'Simulated data, seed {seed}')
print(f'Assumptions: control rate={p_control:.1%}; equal group size; '
      f'two-sided pooled z test; alpha={alpha:.2f}; '
      f'{replicates:,} runs per case')
print('n per arm | power for 6.4% drop | power for 10% drop')
for n in sample_sizes:
    power_64 = simulated_power(n, relative_drops[0])
    power_10 = simulated_power(n, relative_drops[1])
    print(f'{n:>9,} | {power_64:>18.1%} | {power_10:>17.1%}')
false_pos = 1 - (1 - alpha) ** 6
print(f'Six separate null tests at alpha={alpha}: '
      f'{false_pos:.1%} chance of one or more false positives')

Printed output:

Simulated data, seed 20261029
Assumptions: control rate=2.5%; equal group size; two-sided pooled z test; alpha=0.05; 100,000 runs per case
n per arm | power for 6.4% drop | power for 10% drop
   10,000 |              11.5% |             21.2%
   20,000 |              18.0% |             37.6%
   40,000 |              31.6% |             64.1%
   60,000 |              43.9% |             81.2%
Six separate null tests at alpha=0.05: 26.5% chance of one or more false positives

Assumptions for the table: 2.5% buy rate in the control group, equal group sizes, a two-sided pooled z test, alpha 0.05, 100,000 simulated runs per row.

Result: At 60,000 per group, power is 43.9% for a 6.4% drop and 81.2% for a 10% drop. Power is the chance that a test finds an effect. At this group size the test finds the smaller drop in fewer than half of the runs. If all six tests are separate, all null, and each uses alpha 0.05, the chance of one or more false positives is 26.5%. The result shifts if tests are linked.

Limit: The model assumes a steady buy rate and separate visitors. It does not model shifts over time, repeat visits, or group effects.

Sources

• Wasserstein, R. and Lazar, N. (2016), "The ASA Statement on p-Values: Context, Process, and Purpose," The American Statistician, doi 10.1080/00031305.2016.1154108. Cited for one point: a p-value alone does not measure effect size or importance.
• Benjamini, Y. and Hochberg, Y. (1995), "Controlling the False Discovery Rate," Journal of the Royal Statistical Society Series B, doi 10.1111/j.2517-6161.1995.tb02031.x. Cited for one point: methods exist to control errors across many tests.

Power Plan Risks

Before we discuss what power analysis is, let's start with what can go wrong.

Have results from your own experiment? Upload them to the free A/B test tool and see whether any variant beat the control, and by how much.

Mistake #1: Using Generic "Small/Medium/Large" Effect Sizes

Generic labels for small, medium and large effects do not say whether an effect matters to your business. Practitioners often plug a standard number into a power calculator without asking that question. Only your business model can say whether a given difference is worth the weeks of engineering time a test takes. What did we believe before seeing this data? If you expected a 50% lift from a major redesign, planning to detect a much smaller lift spends resources on precision you don't need.

Mistake #2: Ignoring Prior Beliefs About Effect Size

Traditional power analysis treats the effect size as a known constant. But in reality, we're uncertain about the effect before running the experiment. That's why we're testing. This uncertainty matters for planning.

Imagine you're testing a checkout page redesign. Your prior beliefs might be: "Based on similar tests, we expect somewhere between a 5% and 20% lift, most likely around 12%." A Bayesian approach would model this uncertainty and calculate the probability of obtaining convincing evidence across the plausible range of effects.

The frequentist power calculation assumes the effect is exactly (say) 12%, then asks: "What's our probability of rejecting the null hypothesis?" But if the true effect is 8%, your actual power is lower. If it's 16%, your power is higher. The single power value (e.g., "80% power") hides this uncertainty.

Mistake #3: Calculating Power After Seeing Results

You run an experiment. It's not significant (p = 0.18). Someone asks: "What was our power?" You calculate post-hoc power using the observed effect size. It's 35%. You conclude: "We were underpowered, so we can't rule out an effect."

This reasoning is circular. Post-hoc power calculated from the observed effect mostly restates the p-value. It adds little new information. If p = 0.18, you already know the evidence was weak. Calculating power doesn't change that.

The posterior distribution tells a richer story than a single number. Instead of post-hoc power, calculate a confidence interval for the effect size. If your 95% CI is [-2%, +8%], you can say: "We're 95% confident the true effect is between a 2% decrease and an 8% increase. Effects larger than 8% are unlikely given this data."

Mistake #4: Treating Power as a Pass/Fail Threshold

Many teams set a rigid "80% power" requirement without understanding what it means. Power is the probability of detecting an effect if it exists at the specified size. At 80% power, you still have a 20% chance of a false negative (Type II error), failing to detect a real effect.

The 80% convention comes from balancing Type I errors (false positives, typically set at 5%) with Type II errors. It implies you consider false negatives 4× more tolerable than false positives. Is that true for your decision?

In medical studies testing a potential cancer treatment, you might want 90% or 95% power: false negatives (missing an effective treatment) have severe consequences. In low-risk product experiments, 70% power might be acceptable if running larger samples is prohibitively expensive.

Mistake #5: Ignoring Multiple Comparisons

You design an experiment with 80% power to detect a 10% lift in your primary metric (conversion rate). But you're also tracking five secondary metrics: average order value, items per cart, bounce rate, time on site, and return rate.

If all six tests are separate, all null, and each uses α = 0.05, the chance of one or more false positives is 26.5%, as the example above shows. The result shifts if tests are linked. Testing more metrics can raise the chance of at least one false positive. A fix for many tests can lower power for each metric. A larger sample may help make up for that loss.

This becomes worse when you're running multiple A/B tests simultaneously, each with multiple metrics, and reporting whichever shows significance. The garden of forking paths multiplies your false positive risk.

What Power Analysis Actually Tells You (And What It Doesn't)

Power analysis answers a specific question: "If I run this experiment with N observations, and the true effect size is exactly X, what's my probability of obtaining a statistically significant result at significance level α?"

Four quantities are mathematically linked:

• Sample size (N): How many observations per group
• Effect size (δ): The standardized difference you're trying to detect
• Significance level (α): Your false positive tolerance (typically 0.05)
• Power (1-β): Probability of detecting the effect if it exists

Specify any three, and you can calculate the fourth. Most commonly, you specify effect size, α, and desired power (say, 80%), then solve for required sample size.

What power analysis doesn't tell you:

• Whether the effect exists: it assumes a specific effect size and asks about detection probability
• The probability your hypothesis is true: that's a Bayesian posterior, not a frequentist power calculation
• How much evidence you'll obtain: power is about crossing a significance threshold, not quantifying evidence strength
• What to conclude from non-significant results: even high-powered studies can miss effects by chance

How to Actually Run Power Analysis (The Right Way)

Here's a principled workflow that avoids the common pitfalls:

Step 1: Define Your Minimum Detectable Effect from Business Requirements

Don't start with statistics. Start with decisions. Convene your stakeholders and ask:

• "What improvement would justify the cost of implementing this change?"
• "Below what threshold would we consider the effect too small to matter?"
• "What effect size would change our strategic direction?"

Convert these answers to statistical effect sizes. For conversion rates, that's typically a percentage point difference or relative lift. For continuous metrics (revenue, session duration), you'll need to estimate standard deviations from historical data.

Step 2: Quantify Your Prior Uncertainty

Before seeing data from this experiment, what do you believe about the likely effect size? Sources of prior information:

• Historical data: Previous A/B tests on similar changes
• Industry benchmarks: Published effect sizes from comparable experiments
• Expert judgment: Product team intuitions based on experience
• Pilot studies: Small-scale tests or user research findings

Express this as a range, not a point estimate. "We believe the effect is between 5% and 20%, most likely around 12%" is more honest than "We assume a 12% effect."

Step 3: Calculate Power Across Your Prior Range

Create a power curve showing detection probability for effect sizes spanning your prior belief interval. Here's example code for a two-proportion test:

import numpy as np
from scipy.stats import norm

def power_two_proportions(n_per_group, p1, p2, alpha=0.05):
    """Calculate power for a two-sided two-proportion z-test"""
    # Pooled proportion under null hypothesis
    p_pooled = (p1 + p2) / 2

    # Standard error under null and alternative
    se_null = np.sqrt(2 * p_pooled * (1 - p_pooled) / n_per_group)
    se_alt = np.sqrt((p1 * (1 - p1) + p2 * (1 - p2)) / n_per_group)

    # Cut-offs for the difference in rates, set from the null
    z_crit = norm.ppf(1 - alpha / 2)
    difference = p2 - p1
    lower = -z_crit * se_null
    upper = z_crit * se_null

    # Tail chances under the alternative
    power = (norm.cdf((lower - difference) / se_alt)
             + norm.sf((upper - difference) / se_alt))
    return power

# Example: baseline conversion 3%, testing for 3.3% to 3.9%
baseline = 0.03
effect_sizes = np.linspace(0.003, 0.009, 20)  # 10% to 30% relative lift
n_per_group = 5000

for delta in effect_sizes:
    pwr = power_two_proportions(n_per_group, baseline, baseline + delta)
    relative_lift = (delta / baseline) * 100
    print(f"Relative lift: {relative_lift:.1f}% → Power: {pwr:.2%}")

This reveals detection probability across your uncertainty range, not just at a single assumed effect.

Step 4: Make Sample Size Decisions Under Resource Constraints

Reality check: you probably can't afford the sample size that gives 95% power for your smallest effect of interest. Now what?

You have three principled options:

1. Accept lower power for small effects: "We have 90% power for effects above 15%, but only 60% power for a 10% effect. We're willing to risk missing small effects."
2. Focus on larger effects: "Given our sample constraints, we can reliably detect 20%+ effects. We'll treat this as an initial screen, smaller effects would require follow-up testing."
3. Use Bayesian methods: "Rather than binary significance testing, we'll calculate posterior probabilities and credible intervals. A non-significant result will still provide evidence about plausible effect sizes."

What you shouldn't do: run an underpowered test, get p = 0.12, and announce "we found no effect." Your test may simply have been incapable of detecting the effect that exists.

Step 5: Preregister Your Analysis Plan

Before collecting data, document:

• Primary outcome metric(s)
• Planned sample size and stopping rule
• Significance level and power
• Minimum detectable effect
• Any subgroup analyses you'll run

This prevents post-hoc rationalization ("we were really testing secondary metric Y, not primary metric X") and selective reporting of whichever metric happened to reach significance.

When Sample Size Calculations Lie to You

Standard power analysis formulas make assumptions that often don't hold in real business contexts. Here's when to be skeptical of the numbers:

Time-Based Effects and Seasonality

Your power calculation says you need 10,000 observations per group. At 500 visitors per day, that's 40 days of testing. But your product has weekly seasonality, weekday behavior differs from weekends. And you're testing during November, which includes Black Friday.

The effective sample size may be much smaller than raw observation counts. Observations separated by one day are more correlated than observations separated by one week. Standard formulas assume independent observations, violated when temporal correlation exists.

How much should this evidence update our beliefs? Less than uncorrelated samples would suggest. The needed change depends on how much the data move together. Use past data to size it, or use a method that accounts for time or group effects.

Early Stopping and Sequential Testing

You plan a 4-week test with 80% power. After week 1, you peek at the results. They're significant! You stop early and declare victory. Repeated checks can raise the false positive rate above the planned level if each check uses the same cutoff.

The size of the rise depends on how many checks you make and when. Use a method with cutoffs set for those checks.

If you plan to check results early, set the checks and the cutoffs before the test starts, and size the sample with that plan in mind.

Heterogeneous Treatment Effects

Your average treatment effect might be 8%, but that average masks important variation. Mobile users: +15%. Desktop users: -2%. Power analysis for the average effect doesn't tell you whether you can detect these subgroup differences.

Subgroup analysis requires much larger samples. If you split your sample in half to analyze two subgroups separately, each subgroup has half the observations, so power for each is lower, and lower still if the effect in the subgroup is smaller. Detecting an interaction effect (differential response across subgroups) generally requires a larger sample than detecting a main effect.

If subgroup analysis matters to your decision, power your study for those comparisons, not just the overall average treatment effect.

The Bayesian Alternative: Assurance and Precision

Traditional power analysis has a frequentist framing: "If I repeat this experiment infinite times under the assumption that the effect is exactly δ, what proportion of those repetitions will yield p < 0.05?"

That's useful, but it's not the only way to plan sample sizes. Bayesian approaches offer complementary perspectives:

Assurance: Average Power Over Prior Uncertainty

Instead of calculating power for a single assumed effect size, calculate the expected power averaging over your prior distribution of plausible effect sizes. This is called assurance (or expected power).

If you believe the effect is somewhere between 5% and 15% (uniform prior), assurance is the average power across that range. If you're 80% confident the effect is above 10%, weight the power calculation accordingly.

Assurance accounts for your genuine uncertainty about the effect size, giving a more realistic picture of detection probability than point-estimate power.

Precision-Based Sample Size: Width of Credible Intervals

Instead of asking "Can I reject the null?", ask "How precisely can I estimate the effect size?" Plan sample size to achieve a desired posterior credible interval width.

For example: "I want my 95% credible interval for the conversion rate difference to be no wider than ±1 percentage point." Solve for the sample size that achieves this precision on average (or with 90% probability, if you want to be conservative).

This approach decouples sample size planning from arbitrary significance thresholds. Even if your result isn't "significant," you've still learned something precise about plausible effect sizes.

Decision-Theoretic Sample Size

The most principled Bayesian approach: specify loss functions for different decision errors, then choose sample size to minimize expected loss.

For a price-rise test like the one above:

• Implement price increase when it helps: Gain $X per week
• Implement price increase when it hurts: Lose $Y per week
• Don't implement when it would help: Forgo $X per week
• Don't implement when it would hurt: Avoid losing $Y per week

Combine these payoffs with your prior beliefs and the evidence you'd obtain from different sample sizes. The optimal sample size maximizes expected utility.

This is mathematically elegant but requires specifying explicit loss functions, difficult in practice. Most teams find assurance or precision-based planning more tractable.

Quick Reference: Power Analysis Checklist

When Power Analysis Isn't the Right Tool

Power analysis assumes you're doing frequentist hypothesis testing with a binary decision rule. That's not always appropriate:

Exploratory Research

If you're in early-stage product development trying to understand user behavior, hypothesis testing may be premature. You don't yet know what to test. Descriptive analysis, user research, and exploratory data analysis are more valuable.

In exploratory contexts, focus on estimation precision (credible interval width) rather than hypothesis testing power.

Continuous Monitoring and Adaptive Experiments

Multi-armed bandit algorithms adaptively allocate traffic to better-performing variants while learning. Traditional fixed-horizon power analysis doesn't apply: the sample size isn't predetermined, and there's no single hypothesis test at the end.

For adaptive experiments, use simulation-based planning: simulate many realizations of the bandit algorithm under different effect size scenarios, and assess regret (lost value from suboptimal allocation) and decision accuracy.

When You Can't Randomize

Power analysis assumes randomized treatment assignment. In observational studies, confounding and selection bias dominate. No amount of statistical power overcomes biased assignment.

If randomization is impossible, invest effort in quasi-experimental design (difference-in-differences, regression discontinuity, instrumental variables, propensity score matching) rather than power calculations for naive comparisons.

Your turn

Bring your own data and the question you actually need answered.

MCP AnalyticsData science by email Send us your data and your question, and we will reply. ds@mcpanalytics.ai

Frequently Asked Questions

What's the difference between power analysis and sample size calculation?

They're two sides of the same equation. Sample size calculation asks: "How many observations do I need to detect this effect?" Power analysis asks: "What's my probability of detecting an effect with this sample size?" Both use the same statistical framework: power, effect size, significance level, and sample size are mathematically linked. Change any one and the others adjust.

Can I do power analysis after collecting data?

Post-hoc power analysis (calculating power after seeing results) is often questioned. If your test was significant, post-hoc power is redundant. You already detected the effect. If your test wasn't significant, low post-hoc power doesn't prove the effect doesn't exist. Instead, use confidence intervals to quantify uncertainty about the effect size, or perform sensitivity analysis to understand what effects you could have detected.

What if I can't afford the sample size power analysis recommends?

You have three principled options: 1) Accept lower power and acknowledge you're running an exploratory study (be honest about detection limits). 2) Focus on larger effects you can actually detect with available data. 3) Use Bayesian methods that quantify evidence strength rather than binary significance testing. What you shouldn't do: run an underpowered study and treat null results as conclusive evidence of no effect.

How do I choose the right effect size for power calculations?

Base it on the minimum effect that matters to your business, not arbitrary "small/medium/large" labels. Ask: "What's the smallest change in conversion rate (or revenue, or retention) that would justify implementing this change?" That's your minimum detectable effect. If historical data exists, use observed effect sizes as priors, but add uncertainty. If you're truly uncertain, calculate power curves for multiple plausible effect sizes rather than picking a single number.

Does power analysis work for Bayesian A/B tests?

Traditional power analysis is a frequentist framework, but similar principles apply in Bayesian settings. Instead of "power to reject the null," you'd calculate the probability of obtaining a credible interval that excludes zero (or your region of practical equivalence). Bayesian approaches can also use simulation-based sample size determination, where you simulate data under different scenarios and assess whether your posterior distributions provide adequate evidence for decision-making.

Connected Techniques for Robust Experiments

Power analysis is one piece of rigorous experiment design. These related techniques strengthen your statistical inference:

• A/B Testing and Statistical Significance: Understanding the hypothesis testing framework that power analysis supports: how to interpret p-values and avoid common significance testing mistakes.
• Confidence Intervals: Moving beyond binary significance to quantify precision: the Bayesian alternative to power-focused planning prioritizes credible interval width.
• Multiple Testing Correction: When you're testing multiple hypotheses simultaneously, correction methods prevent inflated false positive rates, critical for accurate power calculations.
• Sequential Analysis: Methods for peeking at results before experiment completion without inflating Type I error: requires modified power calculations for early stopping.
• Effect Size Estimation: How to translate business requirements into statistical effect sizes, and how to estimate them from pilot data or historical experiments.

Moving from Power Calculations to Principled Decisions

Power analysis is fundamentally about honesty, acknowledging what your experiment can and cannot detect. The five mistakes we started with all involve self-deception: pretending you can detect effects you can't, treating arbitrary thresholds as meaningful, or ignoring the uncertainty inherent in planning.

How much should this evidence update our beliefs? The answer depends on how much evidence you designed your experiment to collect. An underpowered test that finds p = 0.09 provides different information than a well-powered test with the same p-value. In the former case, the data are consistent with a wide range of effect sizes. In the latter, you've meaningfully narrowed the plausible range.

Let's quantify our uncertainty, not hide it. Calculate power curves that reveal your detection probability across plausible effect sizes. Acknowledge when resource constraints force you to accept low power for small effects. Use credible intervals to show what you've learned, even when results aren't "significant."

The posterior distribution tells a richer story than a single number. Whether you use frequentist power analysis or Bayesian assurance calculations, the goal is the same: design experiments that provide adequate evidence for the decisions you need to make. That starts with clear thinking about effect sizes that matter, honest assessment of your prior uncertainty, and transparent reporting of what your data can, and cannot, tell you.

Your stakeholders don't need p-values. They need answers to questions like: "Should we implement this change? What's the likely impact on revenue? How confident should we be in that estimate?" Power analysis ensures you collect enough evidence to answer those questions responsibly. Everything else is just mathematics in service of better decisions.

Related Articles

Part of Collection

All Hypothesis Testing Methods

Compare t-test, ANOVA, chi-square, and more statistical tests, when to use each and how to interpret results.

View all methods →

A/B Testing and Statistical Significance

Master the fundamentals of hypothesis testing and learn how to interpret statistical significance correctly in business experiments.

Read more →

Confidence Intervals Explained

Move beyond point estimates to quantify uncertainty. Learn how confidence intervals provide richer insights than binary significance tests.

Read more →

Multiple Testing Correction Methods

Testing multiple hypotheses? Learn correction methods that control false positive rates without sacrificing too much statistical power.

Read more →

Sequential Analysis and Early Stopping

Peek at experiment results without inflating Type I error. Learn sequential testing methods that maintain statistical validity.

Read more →