Mann-Whitney U Test: When t-Tests Fail

Picture two customer segments compared on revenue per transaction. One segment has a few very large orders. A t-test compares means, and extreme values inflate the variance, so the test can lose power and return a large p-value even when most customers in one segment clearly spend more. The Mann-Whitney U test is a common alternative. It ranks every observation from lowest to highest and works with the ranks instead of the raw values, so one extreme value counts as just one more rank.

This guide states what the test compares, what it assumes, and where it can mislead. It includes a reproducible run on simulated server load times, with the Mann-Whitney result, a t-test on the same data for comparison, and the share of random pairs in which a current load time is higher. The test does not compare means. It asks whether a randomly chosen value from one group tends to be larger than a randomly chosen value from the other. That is a statement about the whole distributions. It is a statement about medians only under an extra assumption, covered below.

Before we draw conclusions from any statistical test, check the design. Were the groups independent? Were units randomly assigned? What assumptions does the chosen test make? Mann-Whitney needs fewer assumptions than a t-test, but it is not assumption-free.

Worked Example (Simulated)

Simulated data, seed 241. The numbers below are made up by a script. They are not from a real server test.

Question: Do page load times tend to be lower with a new server setup than with the current setup?

Inputs: Simulated load times in seconds for two independent groups of 12. The script builds in a difference: the new setup is set to be faster on average. The seed is fixed, so the result can be reproduced. The time 0.73 appears in both groups, so there is one tie across groups.

Method: A two-sided Mann-Whitney U test (SciPy 1.16.1, NumPy 2.3.2). A Welch t-test on the raw times and on log times is shown for comparison. It compares the ranks of the two groups. The reported U for current > new counts pairs where a current time is greater than a new time, and ties count as half. The p-value uses the tie-corrected normal approximation.

import numpy as np
from scipy.stats import mannwhitneyu, ttest_ind

SEED = 241
N = 12
rng = np.random.default_rng(SEED)
current = np.round(rng.lognormal(mean=0.65, sigma=0.65, size=N), 2)
new = np.round(rng.lognormal(mean=0.25, sigma=0.45, size=N), 2)
result = mannwhitneyu(current, new, alternative="two-sided", method="asymptotic")
prob = result.statistic / (N * N)
print(f"Simulated data, seed {SEED}")
print("Question: Are load times lower with the new setup "
      "than with the current setup?")
print(f"Current load times (seconds): {current.tolist()}")
print(f"New load times (seconds): {new.tolist()}")
print(f"n per group: {N}")
print(f"Median current (seconds): {np.median(current):.2f}")
print(f"Median new (seconds): {np.median(new):.2f}")
print(f"Mann-Whitney U for current > new: {result.statistic:.1f}")
print(f"Two-sided p-value (ties corrected): {result.pvalue:.4f}")
print(f"Pairwise probability current > new: {prob:.4f}")
print(f"Mean current: {current.mean():.2f}  Mean new: {new.mean():.2f}")
welch = ttest_ind(current, new, equal_var=False)
print(f"Welch t-test two-sided p-value: {welch.pvalue:.4f}")
logt = ttest_ind(np.log(current), np.log(new), equal_var=False)
print(f"Welch t-test on log times, p-value: {logt.pvalue:.4f}")

Simulated data, seed 241
Question: Are load times lower with the new setup than with the current setup?
Current load times (seconds): [0.73, 4.92, 4.02, 2.72, 1.51, 1.63, 3.82, 0.55, 1.18, 1.9, 1.87, 4.48]
New load times (seconds): [0.96, 1.35, 1.05, 1.05, 0.78, 1.93, 1.41, 2.95, 0.73, 1.74, 1.12, 1.44]
n per group: 12
Median current (seconds): 1.89
Median new (seconds): 1.24
Mann-Whitney U for current > new: 103.5
Two-sided p-value (ties corrected): 0.0734
Pairwise probability current > new: 0.7188
Mean current: 2.44  Mean new: 1.38
Welch t-test two-sided p-value: 0.0387
Welch t-test on log times, p-value: 0.0750

Result: The median was 1.89 seconds for the current setup and 1.24 for the new one. A current time was greater than a new time in about 72% of the 144 pairs. The two-sided p-value is 0.0734, so this run gives limited evidence of a difference. The script builds in a difference, and with 12 per group the test did not clear 0.05. A result that is not clear is not proof of no effect.

Against the t-test: the means were 2.44 and 1.38 seconds. The Welch t-test gave p = 0.0387, which does clear 0.05, and the t-test on log times gave p = 0.0750. So on this one run the rank test was not the more sensitive one. A skewed sample does not guarantee that Mann-Whitney will find more. One simulated run cannot say which test has more power in general. The tests also answer different questions: the t-test compares means, Mann-Whitney compares ranks. Pick the test before you see the p-values (see Mistake 5).

Limit: This is simulated data, not a real server test. It does not show that a new setup will be faster in practice.

Sources

• Mann H, Whitney D (1947). On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. Annals of Mathematical Statistics. Cited for who published the test and when.
• Wilcoxon F (1945). Individual Comparisons by Ranking Methods. Biometrics Bulletin. Cited for the earlier rank sum procedure.
• SciPy documentation: scipy.stats.mannwhitneyu. Cited for the function, its tie handling and its exact and asymptotic methods.

Step 1: Diagnose Why Your t-Test Is Invalid

T-tests assume your data follows a normal distribution within each group. A departure from normality alone does not show that a t-test is invalid. Check the data shape and study design too. Before reaching for Mann-Whitney, verify that you actually have a problem.

Check for Normality Violations

Examine your data for these warning signs:

• Skewness: Revenue, conversion time, website load time, and customer lifetime value are often right-skewed. Most values cluster low, with a long tail of high values. If your histogram looks like a hockey stick, you've got skew.
• Outliers: A few extreme observations relative to the bulk of your data. For example, one customer might spend $50,000 while most spend far less. These inflate variance and can reduce t-test power.
• Small samples: Small groups are a reason to take care, not a rule that decides the test. Check the data shape, group sizes, outliers, and study design.
• Bounded data: Response times can't be negative. Conversion rates sit between 0% and 100%. If your data piles up against a natural boundary, it's not normal.

Look at histograms or Q-Q plots of each group. A Shapiro-Wilk test can help (a p-value below 0.05 is a rule of thumb for flagging non-normality), but with large samples it flags tiny departures and with small samples it misses large ones. Judge the shape, not the p-value alone.

Step 2: Understand How Mann-Whitney Ranks Instead of Averages

Mann-Whitney operates on a simple principle: rank all observations from both groups together, then test whether one group occupies systematically higher ranks.

The Ranking Process

Here's a concrete example with website load times (seconds):

Group B has ranks 1, 2, 4, 7. Sum = 14. Group A has ranks 3, 5, 6, 8. Sum = 22. If groups performed identically, we'd expect rank sums near equal (both around 18). Group A's higher rank sum points toward slower load times. With only 4 per group this is far too little data to conclude anything: running the same eight numbers through SciPy gives a two-sided p-value of 0.343 (a Welch t-test gives 0.379). The table shows how ranking works.

The test calculates the U statistic, which counts how many times observations from one group exceed observations from the other. Under the null hypothesis (both groups drawn from the same distribution), U follows a known distribution. Large deviations from the expected U produce small p-values.

Why Ranking Solves the Outlier Problem

Look at that 32.5-second timeout in Group A. In a t-test, this value contributes its full size to the mean and variance, and it dominates the calculation. In Mann-Whitney, it is just rank 8: the highest value, but it would count the same if it were 2.4 seconds.

This is Mann-Whitney's core advantage: it preserves ordinal information (which group tends higher) while discarding the potentially misleading scale information (how much higher in the original units). For skewed data, that's exactly what you want.

Step 3: Decide Between Mann-Whitney, t-Test, and Other Alternatives

Choosing the right test depends on your data's properties and your research question. Here's the decision framework.

Use Mann-Whitney When:

• Data is clearly non-normal: Skewed distributions, outliers present, bounded scales (like time-to-event data that can't go below zero)
• Small samples with doubtful shape: small groups where you cannot check normality and the data look skewed
• Ordinal data: You measured satisfaction on a 1-10 scale, or ranked preferences. The intervals between scale points aren't necessarily equal, so means are questionable
• Robustness is critical: You're unsure about data quality and want a test less sensitive to extreme values

Use a t-Test When:

• Data is approximately normal: Q-Q plots look roughly linear, no obvious skew or outliers
• Reasonable sample sizes: large enough that the mean is stable (use the Welch version if the spreads differ)
• You care about means specifically: Your business question asks "what's the average difference?" not "which group tends higher?"
• Power on normal data: a t-test is somewhat more powerful than Mann-Whitney when the data really are normal. If your data meet the assumptions, the t-test is a good choice

Use Kruskal-Wallis (Not Mann-Whitney) When:

You have three or more groups to compare. Kruskal-Wallis is the non-parametric equivalent of one-way ANOVA. It extends the ranking logic to multiple groups simultaneously. You'll still need post-hoc tests (like Dunn's test) to identify which specific pairs differ.

Consider Transforming Your Data Instead

Before defaulting to Mann-Whitney, try transforming your data to achieve normality. Log transformation often works for right-skewed data like revenue or load times. If log(Y) is approximately normal, run a t-test on the transformed data. A t-test on logs compares geometric means, and back-transforming the difference gives a ratio of geometric means, not a difference in arithmetic means. This keeps the t-test but changes the question, so say so.

However, transformations complicate interpretation ("we found a 0.3 difference in log-dollars"), and they don't always work. If transformation fails or interpretation becomes too convoluted, Mann-Whitney is the cleaner choice.

Step 4: Calculate Required Sample Size Before You Start

What's your sample size? Is this test adequately powered? Small groups give any test little power. Before collecting data, calculate your minimum required n.

Effect Size Matters More Than You Think

Whether a result is significant depends on effect size, sample size, and variability. If the true effect is modest and the study is underpowered, you will often see "no significant difference" even though a real effect exists.

Before starting your experiment, estimate the smallest effect size that would matter for your business decision. Then calculate the required sample size. If you can't achieve that n, reconsider whether the experiment is worth running.

Step 5: Interpret the U Statistic as More Than Just a P-Value

The p-value tells you how surprising the data would be if both groups came from the same distribution. The U statistic, turned into a probability, tells you how often one group exceeds the other. Both matter.

Understanding U as an Effect Size

U counts how many times observations from one group exceed observations from the other group. Transform U into a probability:

If U_A counts pairs in which an A value exceeds a B value, counting ties as half, then P(A > B) = U_A / (n_A × n_B).

In the worked example above, U for current > new is 103.5, so P(current > new) = 103.5 / 144 = 0.72. A current load time was greater than a new load time in about 72% of the 144 pairs, with ties counted as half.

This probability gives a direct reading of how often a value in one group exceeds a value in the other. State which group is counted and how ties are handled.

Converting U to Common Language Effect Size

This probability is sometimes called the "Common Language Effect Size" (CLES) or probability of superiority:

CLES = P(A > B) + 0.5 * P(A = B)

For the worked example, CLES = 0.72 means a current load time was greater than a new load time in about 72% of the 144 pairs. Report it with plain words: "If you pick one page load from each setup at random, the current setup is slower about 72% of the time in this simulated run."

Step 6: Handle Tied Ranks Correctly

Tied values, multiple observations with identical measurements, complicate rank assignment. Software handles them, but check which method it used.

The Tie Correction Procedure

When values tie, assign each the average of the ranks they would occupy. Example: three observations tie for ranks 8, 9, and 10. Assign each rank 9.

Values:  2.1, 2.3, 2.3, 2.3, 2.5, 2.7
Ranks:   1,   2,   3,   4,   5,   6
Tied:    1,   3,   3,   3,   5,   6  (average of 2-4 is 3)

Modern statistical software applies this automatically. It also adjusts the variance calculation to account for ties, which keeps the p-value valid when ties are present.

When Ties Become Problematic

If many of your observations are tied, test power can decrease. This happens with:

• Coarse measurements: Response times rounded to nearest second instead of milliseconds
• Discrete scales: 1-5 satisfaction ratings where many people choose "3"
• Ceiling/floor effects: Many values pile up at the minimum or maximum possible measurement

If ties dominate your data, consider whether your measurement instrument has sufficient precision. Can you measure more precisely? For truly ordinal data (like Likert scales), ties are unavoidable, just ensure you have adequate sample size to compensate for the power loss.

Exact vs. Normal Approximation

For small samples without ties, the exact distribution of U can be used instead of the normal approximation. The exact method assumes no ties, so when values tie across groups, as in the worked example, use the tie-corrected normal approximation. Check which method your software used.

Step 7: Avoid the Most Common Mann-Whitney Mistakes

Mann-Whitney is less sensitive to outliers than a t-test, but it can still be misused. These mistakes can undermine your results.

Mistake 1: Using Mann-Whitney on Paired Data

Mann-Whitney requires independent groups. If you measured the same subjects twice (before/after, left/right), use the Wilcoxon signed-rank test instead. Pairing is a design that can increase statistical power, and an independent-samples test throws that away.

Example: You measure website load time for 50 users on both the current setup and the new setup. Same users, two conditions. That's paired data. Use Wilcoxon signed-rank, not Mann-Whitney.

Mistake 2: Interpreting Results as Differences in Medians

Mann-Whitney is often described as "comparing medians." That holds only when the two distributions have the same shape and differ by a shift. In general, the test responds to whether values in one group tend to be larger than in the other. If the distributions differ in shape (one is more variable or more skewed), a significant result can reflect the shape difference, and two groups with equal medians can still give a small p-value.

Often what you care about is whether one group tends to outperform the other, whichever way. Then say so: "A load time from the new setup tends to be lower than one from the current setup" rather than "The new setup has a lower median load time." If you need a claim about medians, check that the shapes look similar, or compare medians directly, for example with a bootstrap confidence interval.

Mistake 3: Running Multiple Mann-Whitney Tests Without Correction

If you compare five groups pairwise (10 comparisons), your chance of at least one false positive is no longer 5%. If the 10 tests were independent and every null hypothesis were true, it would be 1 - 0.95^10, about 40%. Pairwise tests share groups, so the exact figure differs. Apply multiplicity corrections (Bonferroni, Holm, or FDR) or use Kruskal-Wallis followed by post-hoc tests designed for multiple comparisons.

Before you start testing, decide how many comparisons you'll make. Correct your α threshold accordingly. This is experimental design, not post-hoc data mining.

Mistake 4: Ignoring Unequal Sample Sizes

Mann-Whitney works with unequal group sizes, but power decreases as imbalance increases. If n₁ = 100 and n₂ = 10, you'll have less power than balanced groups totaling 110. Aim for equal or near-equal sample sizes when designing experiments.

Mistake 5: Cherry-Picking Tests Until One Shows Significance

You run a t-test: p = 0.09. Not significant. You try Mann-Whitney: p = 0.04. Significant! You report Mann-Whitney. This is p-hacking. Your decision to use Mann-Whitney must be made before looking at results, based on data properties (normality, outliers, skewness), not on which test gives better p-values.

A causal reading needs a design that supports it, such as random assignment, and an analysis plan fixed before you look at results.

Step 8: Report Results So Non-Statisticians Understand

Your stakeholders don't care about U statistics. They care whether the new setup is worth deploying. Translate statistical results into business language.

Template for Reporting Mann-Whitney Results

Statistical statement: "We compared simulated page load times between the current setup (n = 12) and the new setup (n = 12) using the two-sided Mann-Whitney U test with a tie-corrected p-value (U = 103.5, p = 0.0734). A current time was greater than a new time in about 72% of the 144 pairs. The difference did not clear 0.05 in this run."

Plain English translation: "In this simulated run, the new setup looked faster, but the evidence was limited. With 12 loads per group, a result that is not clear is not proof of no effect."

What to Include in Every Report

• Sample sizes: n for each group
• Test statistic: U value
• P-value: With interpretation (report the exact p-value, and say what threshold you chose before the test)
• Effect size: Common Language Effect Size (probability one group exceeds the other)
• Direction: Which group is higher/better/faster
• Practical significance: Does this difference matter for decisions?

Visualization Matters

Include box plots or violin plots showing both distributions. These communicate more than any p-value. Stakeholders will see the overlap, the medians, the outliers, and the overall pattern. A picture clarifies what "72% of pairs" means in concrete terms. This page does not include a chart of the worked example.