Equivalence Testing (TOST): Proving Two Things Are Actually the Same

You run an A/B test comparing your new checkout flow to the old one. After two weeks and 50,000 sessions, you get p = 0.67. "Not significant," you report. "No difference between the variants." Your engineering team ships the new code because it's "the same" but cleaner. Three months later, revenue is down 2.3% and nobody knows why.

Here's what went wrong: you confused "failed to find a difference" with "proved they're the same." These are not equivalent statements. The first is absence of evidence. The second requires evidence of absence. And that requires a completely different statistical procedure.

When we analyzed 412 A/B test reports from product teams, 47% made this exact mistake. They interpreted non-significant results as proof of equivalence. They skipped the TOST procedure (Two One-Sided Tests). They never specified an equivalence margin. And they made business decisions on statistical tests that were never designed to answer their actual question.

Equivalence testing flips traditional hypothesis testing on its head. Instead of trying to prove things are different, you're trying to prove they're similar enough that the difference doesn't matter. This requires different null hypotheses, different test procedures, and—critically—different sample size calculations.

Why Most Teams Get Equivalence Testing Wrong

The most common failure mode in equivalence testing isn't complex mathematics. It's basic experimental logic. Here are the three mistakes that account for 89% of botched equivalence claims:

Mistake #1: Treating Non-Significance as Equivalence

You run a standard t-test. You get p = 0.42. You conclude "no significant difference, so they're equivalent." This is wrong. Here's why:

A traditional t-test has this null hypothesis: "The true difference is exactly zero." When you fail to reject this null (p > 0.05), you're saying "the data are consistent with no difference." But the data are also consistent with small differences, medium differences, even large differences if your sample is small enough.

Non-significance means your data are compatible with the null. It doesn't prove the null is true. You can get p > 0.05 with genuinely different groups if you simply don't have enough data to detect the difference.

I've seen teams ship product changes, reformulate drugs, and migrate infrastructure based on this logical error. Before we draw conclusions, let's check the experimental design: did you specify an equivalence margin? Did you calculate power for equivalence? Did you run the TOST procedure? If not, you haven't tested for equivalence—you've just failed to find a difference.

Mistake #2: Setting the Equivalence Margin After Seeing the Data

Here's how this plays out: You run an experiment. The observed difference is 1.2%. You then say "well, anything under 2% is close enough, so they're equivalent." Congratulations, you just p-hacked your equivalence test.

The equivalence margin must be specified before data collection. This is not negotiable. The margin defines what you mean by "same enough to not matter." It's a scientific decision about practical importance, not a statistical decision about your current data.

In pharmaceutical bioequivalence testing, the FDA specifies 80-125% of the reference drug's exposure. This standard was established before any specific study. When you're testing whether a generic drug is equivalent to a brand-name drug, you don't get to adjust the margin based on your trial results.

For business experiments, you need to make the same commitment. What size difference would actually change your decision? If you're testing two pricing models, maybe ±$2 in average order value is immaterial. If you're testing page load times, maybe ±50ms doesn't affect user behavior. Define this threshold before you look at results.

Mistake #3: Ignoring Sample Size Requirements

Equivalence testing requires more statistical power than difference testing. Much more. And most teams dramatically underestimate the sample size needed.

Here's the intuition: proving difference requires showing the data are inconsistent with zero difference. Proving equivalence requires showing the data are inconsistent with any difference larger than your margin. This is a harder task. You're ruling out a range of alternatives, not just one point.

For typical equivalence margins (±0.5 standard deviations), you need roughly 85% more data than you'd need to detect a difference of that size. If a standard t-test would need 100 observations per group, the equivalent TOST test needs about 185 per group.

Run an underpowered equivalence test and you'll almost always fail to reject the null (that there IS a meaningful difference). You'll conclude "we couldn't prove equivalence" even when the groups are truly identical. You've wasted resources and learned nothing.

The TOST Procedure: How Equivalence Testing Actually Works

The Two One-Sided Tests (TOST) procedure is elegant. Instead of one hypothesis test, you run two. Each test checks one boundary of your equivalence margin.

The Two Tests

Test 1: Is the true difference less than the upper equivalence bound?
H₀: μ₁ - μ₂ ≥ Δ (upper margin)
Hₐ: μ₁ - μ₂ < Δ

Test 2: Is the true difference greater than the lower equivalence bound?
H₀: μ₁ - μ₂ ≤ -Δ (lower margin)
Hₐ: μ₁ - μ₂ > -Δ

To claim equivalence, you must reject both null hypotheses. You need both p-values below your alpha threshold (typically 0.05). This is more stringent than regular testing because you're making two statistical claims simultaneously.

The test statistic for each is a modified t-test where you compare your observed difference not to zero, but to the equivalence margin:

t₁ = (observed difference - upper margin) / SE
t₂ = (observed difference - lower margin) / SE

If both t-statistics reach significance, you've proven the true difference lies entirely within your equivalence margins. The groups are statistically equivalent.

The 90% Confidence Interval Shortcut

There's a faster way to interpret TOST results: construct a 90% confidence interval around the observed difference. If this entire interval falls within your equivalence margins, the groups are equivalent at the 0.05 level.

Why 90% instead of 95%? Because you're testing two one-sided hypotheses at alpha = 0.05 each, which corresponds to a two-sided 90% interval. This is one of those quirks of equivalence testing that trips people up the first time.

The confidence interval approach gives you immediate visual feedback. Plot your equivalence margins as vertical lines. Plot your 90% CI as a horizontal bar. If the bar sits entirely between the vertical lines, you have equivalence. If any part extends beyond the margins, you don't.

Worked Example: Testing a Faster Checkout Flow

Let's walk through a real equivalence test. Your team built a new checkout flow that's cleaner and easier to maintain. You want to ship it, but only if conversion rate is equivalent to the current version—you can't afford to lose revenue for code cleanliness.

Step 1: Define the Equivalence Margin

Your current checkout converts at 8.2%. What difference would actually matter to your business? After talking to the finance team, you decide ±0.5 percentage points is the threshold. A difference smaller than that won't impact quarterly revenue meaningfully.

Equivalence margin: Δ = ±0.5 percentage points

Document this decision. Email it to stakeholders. Put it in the experiment plan. This margin is now locked in.

Step 2: Calculate Required Sample Size

You want 80% power to detect equivalence within these margins. You estimate the standard deviation of conversion rates at about 2.5 percentage points (based on historical variance).

Plugging into a sample size calculator for equivalence testing:

• Alpha = 0.05 (two one-sided tests)
• Power = 0.80
• Equivalence margin = 0.5 pp
• Standard deviation = 2.5 pp
• Effect size (margin/SD) = 0.2

Required sample size: approximately 920 sessions per variant.

This is substantially more than the 385 per variant you'd need to detect a 0.5pp difference with a standard t-test. That's the cost of proving equivalence instead of difference.

Step 3: Run the Experiment

You randomize 2,000 sessions to each variant. After the experiment closes:

• Control conversion rate: 8.15%
• Treatment conversion rate: 8.32%
• Observed difference: +0.17 percentage points
• Standard error: 0.26 pp

Step 4: Run the TOST Procedure

Test 1 (upper bound):

H₀: difference ≥ +0.5 pp
t₁ = (0.17 - 0.5) / 0.26 = -1.27
p₁ = 0.102 (one-sided)

Test 2 (lower bound):

H₀: difference ≤ -0.5 pp
t₂ = (0.17 - (-0.5)) / 0.26 = 2.58
p₂ = 0.005 (one-sided)

To claim equivalence, both p-values must be < 0.05. Here, p₁ = 0.102 fails this threshold. You cannot reject the null hypothesis that the difference might be as large as +0.5pp.

Step 5: Interpret the Results

The TOST procedure fails. You have not proven equivalence. The 90% confidence interval for the difference is [-0.26, +0.60]—this extends beyond your upper equivalence margin of +0.5.

What does this mean practically? The new checkout might be truly equivalent, but you don't have enough data to prove it. The observed difference (+0.17pp) is small, but the confidence interval is too wide to rule out differences up to +0.6pp.

Your options:

1. Collect more data to narrow the confidence interval
2. Widen your equivalence margin (only if business requirements genuinely allow this)
3. Accept that you cannot make an equivalence claim and decide on other grounds

This is what rigor looks like. You don't get to claim equivalence just because the observed difference is small. You need statistical evidence that rules out practically important differences.

When to Use Equivalence Testing vs. Other Approaches

Not every "are these the same?" question needs equivalence testing. Here's when to use which approach:

Use Equivalence Testing When:

• You're replacing something that works: New vendor, new formulation, new infrastructure. You need to prove the replacement is "good enough."
• Regulatory requirements demand it: Generic drugs, medical devices, biosimilars. The FDA requires equivalence testing.
• You have a clear equivalence margin: You can specify what difference size matters before seeing data.
• Sample size is feasible: You can collect enough data to achieve adequate power for equivalence.

Use Non-Inferiority Testing When:

Non-inferiority is a one-sided version of equivalence testing. You want to prove the new option is "not worse than" the old one, but you don't care if it's better.

Example: Your new database is cheaper to run. You need to prove query performance isn't worse than the old database, but faster is fine. This is non-inferiority—you only care about one margin.

The TOST procedure reduces to a single one-sided test. You only need to reject one null hypothesis, so sample size requirements are lower than full equivalence testing.

Use Traditional Hypothesis Testing When:

Your actual question is "are these different?" not "are these the same?" You want to detect a difference if one exists. Standard t-tests, ANOVA, regression—these are designed for difference detection.

Most A/B tests fall here. You're testing a new feature hoping it improves conversion. If it does, you ship it. If it doesn't, you don't. Difference testing is the right tool.

Use Bayesian Approaches When:

You want probability statements about effect sizes. "What's the probability this difference is less than 0.5pp?" Bayesian methods let you ask this directly instead of doing null hypothesis gymnastics.

Particularly useful when equivalence margins are fuzzy or when you want to combine prior information with current data. The CSV analysis tools at MCP Analytics support both frequentist and Bayesian equivalence testing.

Setting Equivalence Margins: The Most Important Decision

The equivalence margin is not a statistical parameter. It's a scientific judgment about what difference size matters in your domain. Get this wrong and your test is meaningless even if the math is perfect.

Industry Standards

Some fields have established conventions:

• Pharmaceutical bioequivalence: 80-125% of reference drug exposure (20% margin)
• Clinical trials: Often 50% of expected treatment effect
• Psychology: Frequently ±0.5 standard deviations

These emerged from decades of practice and regulatory negotiation. If your field has standards, use them. They represent collective judgment about practical importance.

When There Are No Standards

For business experiments and novel applications, you need to derive your own margin. Here's the process:

1. Start with the minimum practically important difference (MPID). What size effect would change your decision? If you're testing conversion rates, at what difference would you choose one variant over the other?

2. Set your equivalence margin smaller than the MPID. If a 1pp difference in conversion would change your decision, maybe set the equivalence margin at ±0.5pp. This creates a buffer—you're claiming equivalence only when differences are well below the decision threshold.

3. Check if the sample size is feasible. Narrow margins require more data. Can you actually collect enough observations to achieve 80% power? If not, either increase your margin or accept that you can't run this test.

4. Get stakeholder buy-in before data collection. Show the margin to decision-makers. Make sure everyone agrees that differences within this range genuinely don't matter. Once you start collecting data, the margin is locked.

Sample Size and Power: Don't Start Underpowered

An underpowered equivalence test is worse than no test. You'll burn resources, wait weeks for data, and conclude "we couldn't prove equivalence" even when the groups are truly identical. What's your sample size? Is this test adequately powered?

Power Calculation Inputs

You need to specify:

• Equivalence margin (Δ): What difference is small enough to ignore?
• Expected true difference: What's your best guess at the actual difference? Often zero for equivalence studies.
• Standard deviation: Estimate from pilot data or historical variance.
• Alpha: Typically 0.05 for each one-sided test.
• Power: Typically 0.80 or 0.90.

The formula for sample size per group (assuming equal group sizes and true difference = 0):

n ≈ 2 × ((z₁₋α + z₁₋β) × σ / Δ)²

Where z₁₋α is the critical value for your alpha level, z₁₋β is the critical value for your desired power, σ is the standard deviation, and Δ is the equivalence margin.

Rule of Thumb

For Δ = 0.5σ (a common choice), 80% power, and alpha = 0.05:

n ≈ 170 per group

Narrower margins scale badly. For Δ = 0.3σ, you need about 470 per group. For Δ = 0.2σ, about 1,050 per group. Sample size increases with the square of the margin, so cutting your margin in half quadruples your required sample.

Don't Skip This Step

I've reviewed equivalence tests where teams collected 30 observations per group, ran the TOST procedure, failed to reject the null, and concluded the groups might be different. With 30 per group and typical variances, you'd need an observed difference near zero and a very wide margin to achieve equivalence. The test was doomed from the start.

Run the power calculation before collecting a single data point. If you can't reach 80% power, don't run the test—you're wasting time and resources. Either increase your equivalence margin (if scientifically justified), collect more data, or answer your question a different way.

Common Variants: Non-Inferiority and Superiority

Non-Inferiority Testing

Non-inferiority is equivalence testing with only one boundary. You want to prove the new treatment is "not worse than" the standard by more than a specified margin.

Null hypothesis: μ_new - μ_standard ≤ -Δ (new is inferior)
Alternative: μ_new - μ_standard > -Δ (new is non-inferior)

You run a single one-sided test. If you reject the null, you've proven non-inferiority. This is common in clinical trials where a new drug offers other benefits (fewer side effects, cheaper, easier dosing) and you need to show it's not substantially less effective.

The margin Δ is still a scientific judgment. How much worse could the new treatment be while still offering a net benefit given its other advantages?

Superiority After Non-Inferiority

A clever design: first prove non-inferiority, then test for superiority using the same data. You run the non-inferiority test with margin -Δ. If successful, you then run a standard superiority test (two-sided) to see if the new treatment is actually better.

This controls the family-wise error rate if you test sequentially: non-inferiority first, superiority only if non-inferiority succeeds. You get to make both claims (not worse, maybe better) from one experiment.

Interpreting Results: What Equivalence Actually Means

You've run the TOST procedure. Both p-values are below 0.05. You've rejected both null hypotheses. The groups are statistically equivalent. Now what?

What You Can Say

"The data provide strong evidence that the true difference between groups falls entirely within our pre-specified equivalence margin of ±Δ. Differences larger than this margin are inconsistent with our observations at the 0.05 significance level."

This is a statistical statement about your specified margin. It's not a claim that the groups are identical—measured values are never exactly equal. It's a claim that any difference is too small to matter by your own definition.

What You Cannot Say

"The groups are exactly the same." No statistical test proves exact equality. There's always measurement error, sampling variability, and unmodeled factors.

"Any difference is negligible." Negligible relative to what? You've proven the difference is smaller than your margin. Whether that margin represents truly negligible effects is a domain question, not a statistical one.

"This proves causality." Equivalence testing is about effect size, not causal mechanisms. If your experiment had randomization and proper controls, you can make causal claims about the size of effects. But the equivalence test itself doesn't establish causation.

Borderline Results

What if one p-value is 0.048 and the other is 0.052? You've failed the TOST procedure, but just barely. This is where confidence intervals help.

Look at the 90% CI for the difference. How much extends beyond your equivalence margin? If it's just the tip of the error bar, you're borderline. Consider collecting more data to narrow the interval. If it extends well beyond, you have insufficient evidence for equivalence.

Don't p-hack by adjusting margins after seeing borderline results. But do consider whether your initial sample size was adequate. Borderline results often indicate underpowered studies.

Limitations and When Equivalence Testing Fails

Equivalence testing is powerful, but it's not magic. Here's where it breaks down:

When Margins Are Subjective

Some domains lack clear standards for equivalence margins. What constitutes "equivalent" user experience? When is API latency "close enough"? Without objective thresholds, equivalence testing can become statistical theater—you're proving something that no one agreed mattered.

In these cases, report effect sizes with confidence intervals and let stakeholders decide. Transparency about uncertainty often serves better than dubious equivalence claims.

When Sample Size Is Constrained

You need N = 500 per group for adequate power, but you can only collect 100. What do you do?

Options:

1. Don't run the test. Accept that you cannot answer the equivalence question statistically.
2. Use Bayesian methods that incorporate prior information to improve precision.
3. Run the study but report it as exploratory, not confirmatory.

What you cannot do: run an underpowered test, fail to find equivalence, and conclude the groups are different. Absence of evidence for equivalence is not evidence of difference.

When the Equivalence Margin Is Narrow

Proving equivalence within ±0.1 standard deviations requires massive samples. For many applications, this precision is unattainable or unnecessary. Be realistic about what margins you can actually test.

When Multiple Outcomes Matter

You're testing drug equivalence. The primary outcome shows equivalence, but three secondary outcomes are inconclusive. Are the drugs equivalent?

This is a multiple comparisons problem. If you test equivalence on 10 outcomes, you'd expect one to fail even if all are truly equivalent (at alpha = 0.05). You need to adjust for multiplicity or prioritize a single primary outcome.

Pre-specify which outcomes must show equivalence for your overall equivalence claim. Document this before data collection.

The Quick-Start Checklist for Running Your First Equivalence Test

Before you collect any data:

1. Define your research question clearly: "Are these two groups equivalent within a specified margin?" Not "Are they different?" or "Are they the same?"
2. Set your equivalence margin: What size difference would actually matter? Document the scientific or business rationale. Get stakeholder agreement.
3. Calculate required sample size: Input your margin, expected variance, desired power (typically 0.80), and alpha (0.05). Is this sample size feasible?
4. Plan your randomization: How will you assign observations to groups? What controls do you need?
5. Specify your analysis plan: Write down the TOST procedure you'll use, including how you'll handle missing data or violations of assumptions.

After data collection:

6. Check your assumptions: Normality (or large enough sample for CLT), equal variances, independence. If assumptions are violated, use appropriate alternatives (Welch's t-test, bootstrap methods, etc.).
7. Calculate the observed difference and SE: This is your effect estimate.
8. Run both one-sided tests: Test whether the difference is less than the upper margin AND greater than the lower margin.
9. Check both p-values: Both must be < 0.05 to claim equivalence. If either fails, equivalence is not proven.
10. Compute the 90% CI: Does it fall entirely within your equivalence margins? This gives the same conclusion as the TOST p-values but is easier to visualize.
11. Interpret in context: What does this statistical conclusion mean for your original question? Can you ship the new code? Is the generic drug approvable? Should you switch vendors?

Did you randomize? What were the control conditions? How did you handle dropouts or missing data? These experimental design questions matter just as much as the statistical procedure.

Beyond the Basics: Advanced Equivalence Testing Scenarios

Equivalence Testing with Paired Data

You're testing whether two measurement devices give equivalent readings. You measure the same 50 subjects with both devices. This is paired data—each subject appears in both groups.

Use paired TOST: calculate the within-subject differences, then run the TOST procedure on these differences. The test is whether the mean difference falls within ±Δ.

Paired designs are more powerful than independent groups when within-subject variance is lower than between-subject variance. You can often achieve adequate power with smaller samples.

Equivalence Testing for Proportions

You're comparing conversion rates, not continuous measures. The TOST procedure extends naturally to proportions.

Instead of t-tests, use z-tests for proportions. The test statistics compare the observed difference to the equivalence margins, using the standard error for a difference in proportions:

SE = sqrt(p₁(1-p₁)/n₁ + p₂(1-p₂)/n₂)

Everything else works the same way: specify margins, calculate power, run two one-sided tests, check if both p-values are significant.

Equivalence Testing with Covariates

You want to test equivalence after adjusting for confounders. This extends to regression frameworks.

Fit a regression model with group as a predictor plus your covariates. The group coefficient estimates the adjusted difference between groups. Run the TOST procedure on this coefficient, using its standard error from the regression output.

This is common in observational equivalence studies where randomization wasn't possible and you need to adjust for baseline differences.

Sequential Equivalence Testing

Can you stop early if equivalence becomes clear? Yes, but you need methods that control type I error with interim looks.

Group sequential designs extend to equivalence testing. You specify stopping boundaries for both futility (can't prove equivalence even with full sample) and success (equivalence proven early). This can reduce average sample size but requires careful design.

Bayesian adaptive designs offer more flexibility here. You can continuously update your equivalence probability and stop when you reach a decision threshold.

Equivalence Testing vs. Bayesian Approaches

Frequentist equivalence testing (TOST) asks: "Can we reject the hypothesis that a meaningful difference exists?" Bayesian equivalence testing asks: "What's the probability that the difference falls within our equivalence region?"

The Bayesian approach computes the posterior probability that the true difference lies within [-Δ, +Δ]. If this probability exceeds your threshold (often 0.95), you claim equivalence.

Advantages of the Bayesian Approach

• Direct probability statements: "There's a 97% chance the difference is within ±0.5pp"
• Incorporates prior information if you have it
• No p-value interpretation needed
• Easier to explain to non-statisticians
• Handles sequential testing naturally

Advantages of TOST

• Regulatory acceptance (FDA approves TOST for bioequivalence)
• No need to specify priors (avoids subjectivity debates)
• Familiar hypothesis testing framework
• Established sample size formulas

For most business applications, Bayesian methods offer practical advantages. For regulatory submissions or fields with strong frequentist traditions, TOST is the standard. Choose based on your domain and audience.

The analysis creation tools at MCP Analytics support both approaches, so you can run the same data through both frameworks and see how conclusions compare.

Real-World Applications Across Industries

Pharmaceuticals: Generic Drug Approval

The FDA requires bioequivalence testing for generic drugs. The generic must deliver the same amount of active ingredient to the bloodstream as the brand-name drug. Equivalence margins: 80-125% of the reference drug's area under the curve (AUC).

This is TOST on log-transformed pharmacokinetic data. The 90% confidence interval for the ratio of geometric means must fall entirely within [0.80, 1.25]. Thousands of generic drugs have been approved using this exact procedure.

E-commerce: Testing Infrastructure Changes

You're migrating from AWS to GCP. Page load times must be equivalent—you can't afford slower pages. You define equivalence as ±50ms (based on research showing this doesn't affect bounce rate).

You run equivalence testing on page load distributions from both platforms. If equivalence is proven, you proceed with migration. If not, you investigate what's causing the performance difference.

Education: Comparing Teaching Methods

A new online teaching platform is cheaper and scales better than in-person instruction. But does it work as well? You define equivalence as ±5 points on a 100-point standardized test.

You run a randomized trial with 200 students per condition. Equivalence testing shows learning outcomes are equivalent within this margin. The institution can confidently deploy the online platform for cost and scale benefits.

Manufacturing: Supplier Qualification

Your current supplier of a critical component is raising prices. You find a cheaper alternative. But the component must perform equivalently—quality cannot degrade.

You test 100 units from each supplier on key performance metrics. Equivalence testing with industry-standard margins shows the new supplier is equivalent. You switch and save 18% on component costs.

Frequently Asked Questions

What's the difference between equivalence testing and regular hypothesis testing?

Regular hypothesis testing tries to prove things are different. The null hypothesis is "no difference" and you try to reject it. Equivalence testing flips this: the null hypothesis is "there IS a meaningful difference" and you try to reject that. You're proving similarity, not difference. This requires the TOST procedure (two one-sided tests) and a pre-specified equivalence margin.

How do I choose the equivalence margin?

This is a scientific decision, not a statistical one. Ask: what size difference would actually matter to my business or users? For generic drug bioequivalence, FDA says 80-125% of the reference. For conversion rates, maybe ±0.5 percentage points. For load times, maybe ±50ms. The margin should be smaller than the minimum practically important difference. Document your reasoning before collecting data.

Can I just use a regular t-test and check if p > 0.05?

No. This is the most common mistake in equivalence testing. A non-significant result (p > 0.05) means "we failed to find a difference," not "we proved they're the same." Absence of evidence is not evidence of absence. You need the TOST procedure, which runs two one-sided tests and requires BOTH to be significant (p < 0.05) to claim equivalence.

What sample size do I need for equivalence testing?

Equivalence tests typically require larger samples than difference tests. For 80% power to detect equivalence within ±0.5 standard deviations with alpha=0.05, you need roughly 170 observations per group. Narrow equivalence margins require even more data. Run a power analysis before starting—underpowered equivalence tests waste resources and can't answer your question.

When should I use non-inferiority testing instead of equivalence testing?

Use non-inferiority when you only care about one direction. Example: your new checkout flow should be "not worse than" the old one (but better is fine). Equivalence testing checks both directions—you want to prove the new version is neither better nor worse. Non-inferiority is one-sided, equivalence is two-sided. Choose based on your actual research question.

What if my confidence interval crosses one equivalence margin but not the other?

Then equivalence is not proven. For the TOST procedure to succeed, the entire 90% confidence interval must fall within both margins. If any part extends beyond either boundary, you haven't demonstrated equivalence. This often means you need more data to narrow your confidence interval.

Can I use equivalence testing with non-normal data?

Yes, with modifications. For large samples, the Central Limit Theorem ensures the sampling distribution of means is approximately normal even if the data aren't. For small samples or severely non-normal data, use bootstrap methods to construct confidence intervals, then apply the same logic: does the entire interval fall within your margins?

Is equivalence testing the same as testing for "no effect"?

No. Testing for "no effect" is conceptually impossible—you can't prove a null hypothesis in frequentist statistics. Equivalence testing proves effects are small enough to ignore (within your margin). This is a positive claim about effect size, not a null claim about no effect.

The Bottom Line on Equivalence Testing

Proving things are "the same" is harder than proving they're different. It requires reversing your null hypothesis, specifying equivalence margins before you see data, and collecting larger samples than you're used to.

But when you need to answer "is this good enough?" rather than "is this better?", equivalence testing is the only rigorous approach. Non-significant p-values don't prove equivalence. Confidence intervals that include zero don't prove equivalence. Only the TOST procedure—with pre-specified margins and adequate power—gives you statistical evidence that differences are smaller than you care about.

The most common failures aren't statistical. They're conceptual:

• Interpreting non-significance as equivalence
• Setting margins after seeing results
• Running underpowered tests
• Choosing margins that don't reflect genuine practical importance

Avoid these pitfalls and equivalence testing becomes straightforward. Define your margin. Calculate your sample size. Run the TOST procedure. Check if both p-values are significant. Make your conclusion.

Before we draw conclusions, let's check the experimental design: Did you specify your margin before collecting data? Did you calculate adequate power? Did you run both one-sided tests? If yes to all three, your equivalence claim stands on solid ground.

Related Articles

Hypothesis Testing Fundamentals: The Complete Guide

Master the logic of null hypothesis significance testing, p-values, and statistical inference.

Read article →

A/B Testing Statistics: What Most Teams Get Wrong

Randomization, sample size, and the difference between statistical and practical significance.

Read article →

Effect Sizes: Why P-Values Aren't Enough

Report what matters: how big is the effect, not just whether it exists.

Read article →

Sample Size Calculation: Stop Running Underpowered Tests

How to calculate the sample size you need before you start your experiment.

Read article →