Wilcoxon Signed-Rank Test for Paired Samples

When your data doesn't follow a normal distribution, the Wilcoxon signed-rank test is one option for comparing paired samples, provided the differences are roughly symmetric around their median. Whether you're measuring before-and-after treatment effects, A/B test results, or repeated measurements, understanding how to apply this non-parametric technique and the conventions for reporting it can transform your analytical approach and help you avoid costly mistakes that plague many data analysis projects.

What is the Wilcoxon Signed-Rank Test?

The Wilcoxon signed-rank test is a non-parametric statistical hypothesis test used to compare two related samples or repeated measurements on a single sample. Unlike its parametric counterpart, the paired t-test, this method doesn't assume that your data follows a normal distribution. Frank Wilcoxon published the signed-rank test in 1945 (Biometrics Bulletin).

The test works by calculating the differences between paired observations, ranking these differences by their absolute values, and then applying signs to create a test statistic. This approach makes it particularly robust when dealing with skewed data, ordinal measurements, or datasets containing outliers that would violate the assumptions of parametric tests.

The Mathematics Behind the Test

The Wilcoxon signed-rank test uses the following process:

1. Calculate the difference between each pair of observations
2. Exclude any zero differences from the analysis
3. Rank the absolute values of the differences
4. Apply the original sign (positive or negative) to each rank
5. Calculate the sum of positive ranks (W+) and negative ranks (W-)
6. The test statistic W is the smaller of W+ and W-

For larger sample sizes (typically n > 20), the test statistic approximately follows a normal distribution, allowing for more straightforward p-value calculation. Statistical software packages handle these calculations automatically, but understanding the underlying process helps you interpret results correctly.

import numpy as np
from scipy import stats

seed = 3
rng = np.random.default_rng(seed)
n = 40
before = rng.normal(3.0, 0.8, size=n)
diff = rng.normal(-0.3, 0.5, size=n)
after = before + diff

stat, p = stats.wilcoxon(after, before)
t, tp = stats.ttest_rel(after, before)
n_faster = int((diff < 0).sum())
skew = stats.skew(diff)

print(f"Simulated data, seed {seed}; {n} paired pages")
print(f"Median load time: before {np.median(before):.2f} s, "
      f"after {np.median(after):.2f} s")
print(f"Median change: {np.median(diff):.2f} s; "
      f"{n_faster} of {n} pages got faster")
print(f"Skewness of the differences: {skew:.2f}")
print(f"Wilcoxon signed-rank: W = {stat:.0f}, p = {p:.6f}")
print(f"Paired t-test on the same data: p = {tp:.6f}")

Simulated data, seed 3; 40 paired pages
Median load time: before 2.82 s, after 2.61 s
Median change: -0.30 s; 30 of 40 pages got faster
Skewness of the differences: -0.43
Wilcoxon signed-rank: W = 126, p = 0.000058
Paired t-test on the same data: p = 0.000039

When to Use the Wilcoxon Signed-Rank Test

Choosing the right statistical test is critical for valid inference. The Wilcoxon signed-rank test is appropriate when you have:

Paired or Matched Data

Your data must consist of paired observations or matched samples. Common scenarios include:

• Before-and-after measurements: Patient blood pressure before and after treatment, website conversion rates before and after redesign
• Matched pairs: Comparing test scores between twins, performance metrics between matched competitors
• Repeated measures: Product ratings from the same users across different time periods
• Within-subject designs: Response times under different experimental conditions for the same participants

Non-Normal Distribution

The test excels when your difference scores violate normality assumptions. Indicators include:

• Skewed distributions with long tails
• Presence of significant outliers that shouldn't be removed
• Small sample sizes where normality is difficult to assess
• Paired measurements where you can rank the size of each difference (for purely ordinal data where only the direction is meaningful, a sign test fits better)

When to Choose Alternatives

The Wilcoxon signed-rank test isn't always the best choice. Consider these alternatives:

• Paired t-test: When differences are normally distributed (offers more statistical power)
• Sign test: When you can only determine the direction of differences, not magnitudes
• Mann-Whitney U test: For independent samples rather than paired data
• Friedman test: When comparing more than two related groups

Key Assumptions and Requirements

While the Wilcoxon signed-rank test is less restrictive than parametric alternatives, it still requires certain conditions to be met for valid results:

1. Paired Observations

Each observation in one group must have a corresponding paired observation in the other group. The pairing must be meaningful and established before analysis. Breaking or misidentifying pairs invalidates the test.

2. Independence of Pairs

While observations within pairs are related, different pairs must be independent of each other. For example, measurements from one patient shouldn't influence measurements from another patient.

3. Continuous or Ordinal Scale

The measured variable should be at least ordinal, meaning you can rank the differences. The test works with continuous data, ordinal scales, and even some discrete measurements that can be meaningfully ranked.

4. Symmetry of Differences

An often-overlooked assumption is that the distribution of differences should be approximately symmetric around the median. Severely asymmetric difference distributions may require alternative approaches or data transformation.

Best Practices for Implementation

Following established best practices ensures your analysis meets industry standards and produces reliable, reproducible results.

Pre-Analysis Planning

Before collecting data, establish clear protocols:

• Define hypotheses explicitly: State null and alternative hypotheses (one-tailed or two-tailed) before seeing the data
• Set significance level: The conventional choice is α = 0.05, but adjust based on the consequences of Type I and Type II errors in your context
• Calculate required sample size: Use power analysis targeting 80% power (0.80) to detect your minimum meaningful effect size
• Document pairing criteria: Clearly specify how observations are paired to maintain consistency

Data Quality Checks

Verify data integrity before running the test:

# Python example using pandas
import pandas as pd
import numpy as np

# Check for missing values in paired data
print(f"Missing in Group 1: {df['before'].isna().sum()}")
print(f"Missing in Group 2: {df['after'].isna().sum()}")

# Verify equal sample sizes
assert len(df['before']) == len(df['after']), "Unequal sample sizes"

# Calculate differences
differences = df['after'] - df['before']

# Check for zero differences
zero_count = (differences == 0).sum()
print(f"Zero differences: {zero_count} ({100*zero_count/len(differences):.1f}%)")

# Visualize difference distribution
import matplotlib.pyplot as plt
plt.hist(differences[differences != 0], bins=20)
plt.xlabel('Difference')
plt.ylabel('Frequency')
plt.title('Distribution of Non-Zero Differences')
plt.show()

Running the Test

Most statistical software provides straightforward implementations. Here's how to perform the test in popular platforms:

# Python with SciPy
from scipy.stats import wilcoxon

# Perform two-tailed test
statistic, p_value = wilcoxon(before, after, alternative='two-sided')
print(f"Test Statistic: {statistic}")
print(f"P-value: {p_value}")

# One-tailed test (testing if 'after' is greater than 'before'):
# SciPy's 'greater' asks whether the FIRST argument tends to be larger,
# so pass 'after' first
statistic, p_value = wilcoxon(after, before, alternative='greater')
print(f"One-tailed P-value: {p_value}")

# R implementation
before <- c(125, 130, 142, 138, 155, 148, 162, 145, 133, 140)
after <- c(120, 128, 138, 142, 150, 145, 158, 148, 130, 137)

# Two-tailed test
result <- wilcox.test(before, after, paired = TRUE, alternative = "two.sided")
print(result)

# Extract components
print(paste("Test Statistic V:", result$statistic))
print(paste("P-value:", result$p.value))

Effect Size Calculation

Statistical significance doesn't indicate practical importance. Always calculate effect size to assess the magnitude of differences. The most common effect size for the Wilcoxon signed-rank test is r, calculated as:

r = Z / sqrt(N)

where Z is the standardized test statistic and N is the number of non-zero pairs. Common conventions for interpretation (guides, not cutoffs):

Interpreting Results

Understanding what your results mean in context separates competent analysts from exceptional ones. Here's how to interpret Wilcoxon signed-rank test output:

The P-Value

The p-value is the probability of seeing a result at least as extreme as yours if the null hypothesis were true. For the signed-rank test the null is that the paired differences are distributed symmetrically around zero; reading that as "no difference in medians" additionally assumes the differences are symmetric. Standard interpretation:

• p < 0.05: Statistically significant evidence of a difference (reject null hypothesis)
• p ≥ 0.05: Insufficient evidence to conclude a difference exists (fail to reject null hypothesis)
• p < 0.001: Highly significant result, strong evidence against null hypothesis

However, industry best practices emphasize reporting exact p-values rather than just comparing to a threshold. A p-value of 0.049 and 0.001 both indicate significance, but represent vastly different evidence strengths.

Test Statistic Interpretation

The test statistic (often denoted as W, V, or T depending on software) represents the sum of ranks. While the p-value is more interpretable, understanding the test statistic provides additional insights:

• How extreme the statistic is counts as evidence against the null hypothesis; which direction counts depends on the statistic and on whether your test is one-sided or two-sided
• Values near the expected value under the null hypothesis suggest no real difference
• The direction of the statistic can indicate which group tends to have higher values

Confidence Intervals

While often overlooked, confidence intervals for the median difference provide valuable context. The 95% confidence interval that software reports is for the Hodges-Lehmann estimate, the median of all pairwise averages of the differences, which is close to but not the same as the median difference. If this interval:

• Excludes zero: Consistent with a significant p-value
• Includes zero: Indicates the true difference might be zero
• Contains only practically meaningful values: Suggests actionable insights even if not statistically significant

Common Pitfalls and How to Avoid Them

Even experienced analysts fall into these traps. Recognizing and avoiding them improves analysis quality and credibility.

1. Using the Test with Independent Samples

The most fundamental error is applying the Wilcoxon signed-rank test to independent samples. This test requires paired data. For independent samples, use the Mann-Whitney U test instead.

How to avoid: Always verify the data structure before analysis. Ask: "Is each observation in group A paired with a specific observation in group B?"

2. Ignoring the Symmetry Assumption

While the test doesn't require normal differences, it does assume symmetric distribution of differences around the median. Highly skewed difference distributions can lead to incorrect conclusions.

How to avoid: Create a histogram of differences. If severely asymmetric, consider the sign test or transformation of the original data.

3. Multiple Testing Without Correction

Running multiple Wilcoxon tests on related comparisons inflates Type I error rates. Many tests raise the risk of a false positive. A correction (Bonferroni, Holm-Bonferroni, or a false discovery rate method) can account for many tests.

How to avoid: Apply Bonferroni correction (divide α by number of tests), use Holm-Bonferroni method for more power, or employ False Discovery Rate (FDR) approaches for many tests.

4. Confusing Statistical and Practical Significance

Large samples can produce statistically significant results for trivially small differences. A p-value of 0.001 doesn't mean the effect is important.

How to avoid: Always calculate and report effect sizes. Define minimum meaningful effect sizes before analysis based on domain knowledge and business requirements.

5. Breaking Pairs During Data Cleaning

Removing outliers or missing values from one group without removing the paired observation destroys the paired structure.

How to avoid: When removing an observation due to data quality issues, remove both members of the pair. Track and report the number of pairs removed.

6. One-Tailed vs. Two-Tailed Confusion

Choosing a one-tailed test after seeing the data direction is a form of p-hacking that invalidates results.

How to avoid: Specify one-tailed or two-tailed in your pre-analysis plan. Use two-tailed tests unless you have strong a priori reasons and would interpret results in only one direction.

Worked Example: E-Commerce Conversion Before and After a Redesign

Illustrative data. The 25 categories and their conversion rates below are invented to demonstrate the method; they are not from a real company. The script runs as written and the output below came from Python with SciPy 1.16.3 and pandas 2.3.3.

Business Context

Suppose an e-commerce company wants to evaluate whether a new product page design is followed by higher conversion rates. It measures the weekly conversion rate for each of 25 product categories before and after the redesign. Pairing by category removes differences between categories (price point, typical demand) from the comparison. It does not remove anything that changed over time for every category at once, such as seasonality or a promotion, so a before-and-after result like this supports "conversion rose after the redesign" and needs a concurrent control group before it can be called an effect of the redesign.

Data Characteristics

• Sample size: 25 product categories (paired observations)
• Outcome variable: conversion rate (percentage, continuous)
• Pairing: each category measured before and after the redesign
• Ties: the rates are recorded to one decimal, so the 25 differences take only three values (0.1 in 4 categories, 0.2 in 7, 0.3 in 14). Heavy ties mean the p-value comes from a normal approximation rather than an exact calculation.

Analysis Steps

import pandas as pd
import numpy as np
from scipy.stats import wilcoxon

# Sample data (conversion rates as percentages)
data = {
    'category': ['Electronics', 'Clothing', 'Books', 'Home', 'Sports',
                 'Toys', 'Beauty', 'Food', 'Garden', 'Automotive',
                 'Jewelry', 'Shoes', 'Music', 'Movies', 'Games',
                 'Office', 'Pet', 'Baby', 'Health', 'Tools',
                 'Outdoor', 'Art', 'Crafts', 'Industrial', 'Luggage'],
    'before': [2.3, 3.1, 1.8, 2.7, 2.2, 3.4, 2.9, 1.9, 2.1, 1.7,
               4.2, 3.8, 2.4, 2.6, 3.3, 2.0, 2.5, 2.8, 2.2, 1.6,
               2.4, 2.1, 1.9, 1.5, 2.7],
    'after':  [2.5, 3.4, 2.0, 3.0, 2.3, 3.7, 3.2, 2.1, 2.4, 1.8,
               4.5, 4.1, 2.7, 2.8, 3.6, 2.2, 2.8, 3.1, 2.5, 1.7,
               2.7, 2.3, 2.1, 1.6, 3.0]
}

df = pd.DataFrame(data)

# Differences, rounded to the one decimal the data is recorded in
# (avoids floating-point noise splitting values that are really tied)
df['difference'] = (df['after'] - df['before']).round(1)

# One-sided test: is 'after' greater than 'before'? Testing the differences
# (after - before) against zero keeps the direction unambiguous.
statistic, p_value = wilcoxon(df['difference'], alternative='greater')

# Effect size r = Z / sqrt(n), using the normal approximation (no tie correction)
n = len(df)
z_score = (statistic - n*(n+1)/4) / np.sqrt(n*(n+1)*(2*n+1)/24)
effect_size = abs(z_score) / np.sqrt(n)

print(f"Sample size: {n} paired categories")
print(f"Categories that improved: {(df['difference'] > 0).sum()} of {n}")
print(f"Median conversion before: {df['before'].median():.2f}%")
print(f"Median conversion after: {df['after'].median():.2f}%")
print(f"Median difference: {df['difference'].median():.2f}%")
print(f"\nTest Statistic (sum of positive ranks): {statistic}")
print(f"P-value (one-tailed): {p_value:.2e}")
print(f"Effect size (r): {effect_size:.2f}")

Results Interpretation

Sample size: 25 paired categories
Categories that improved: 25 of 25
Median conversion before: 2.40%
Median conversion after: 2.70%
Median difference: 0.30%

Test Statistic (sum of positive ranks): 325.0
P-value (one-tailed): 3.75e-06
Effect size (r): 0.87

• Direction is unanimous: all 25 categories improved, so the sum of positive ranks is its maximum possible value, 25 x 26 / 2 = 325, and the sum of negative ranks is 0.
• P-value: about 4 in a million (one-tailed, normal approximation). With a result this lopsided the exact digits matter little; the point is that no plausible chance explanation produces 25 improvements out of 25 in a paired comparison.
• Effect size: r = 0.87, which is large by the usual conventions. The median change is small in absolute terms (0.30 percentage points, 2.40% to 2.70%), so whether it matters commercially depends on traffic and margin, which the test cannot tell you.

What the Test Does Not Tell You

The analysis shows that conversion was higher after the redesign in every category. It does not show the redesign caused it. A concurrent comparison, such as showing the old page to a random share of visitors during the same weeks, would separate the redesign from seasonality and other changes. Also check whether the gain is spread across categories or concentrated in a few, and weigh the result against implementation cost and longer-term effects.

Best Practices for Reporting Results

Clear communication of statistical results ensures stakeholders understand implications and can make informed decisions. Follow these industry-standard reporting practices:

Essential Components

Every Wilcoxon signed-rank test report should include:

1. Sample description: Number of pairs, number of zero differences excluded, percentage of ties
2. Descriptive statistics: Median (and quartiles) for both groups and for differences
3. Test details: Test statistic value, exact p-value, one-tailed or two-tailed
4. Effect size: r value with interpretation
5. Confidence interval: For the median difference, typically 95% CI
6. Practical interpretation: What the results mean in business terms

Example Report Template

A well-structured results section might read:

"We compared conversion rates before and after the redesign across 25 product categories using the Wilcoxon signed-rank test. The median conversion rate increased from 2.40% (IQR: 2.05-2.85%) before redesign to 2.70% (IQR: 2.25-3.15%) after redesign, representing a median increase of 0.30 percentage points (95% CI: 0.15-0.45%).

The Wilcoxon signed-rank test revealed a statistically significant improvement (W = 312, p < 0.001, one-tailed, n = 25 pairs). The effect size was large (r = 0.52), indicating substantial practical significance. No pairs had zero differences, and ties were minimal (8% of comparisons).

These results provide strong evidence that the new design meaningfully improves conversion rates across product categories. At our current traffic levels, this improvement would generate an estimated $2.4M in additional annual revenue."

Visualization Best Practices

Complement statistical results with clear visualizations:

• Before-after plots: Show individual pairs with connecting lines to illustrate changes
• Difference histogram: Display the distribution of differences to show symmetry and spread
• Box plots: Compare distributions before and after in side-by-side plots
• Effect size visualization: Use forest plots or similar for effect sizes with confidence intervals

Related Statistical Techniques

The Wilcoxon signed-rank test is part of a broader family of statistical methods. Understanding related techniques helps you choose the right tool for each situation.

Paired T-Test

The parametric equivalent of the Wilcoxon signed-rank test. Use when differences are normally distributed. Offers more statistical power under normality but less robust to violations. See our comprehensive t-test guide for detailed comparison.

Mann-Whitney U Test

The non-parametric test for independent samples. Use when you have two separate groups rather than paired observations. Tests whether one group tends to have larger values than another.

Sign Test

A simpler non-parametric test for paired data that only considers the direction of differences, not magnitudes. More appropriate when you can only determine whether values increased or decreased, but the amount of change isn't meaningful or reliable.

Friedman Test

Extension of the Wilcoxon signed-rank test for more than two related groups. Useful for repeated measures designs with three or more time points or conditions.

Bootstrapping Methods

Modern resampling approaches that can estimate confidence intervals and test hypotheses without distributional assumptions. Increasingly popular in industry as computational power becomes cheaper.

Advanced Considerations and Extensions

For analysts working with complex data or specialized applications, these advanced topics extend the basic Wilcoxon framework:

Handling Large Datasets

With very large samples (n > 1000), even trivial differences become statistically significant. Focus shifts entirely to effect sizes and confidence intervals. Consider whether median differences meet pre-specified minimum meaningful thresholds rather than simply testing for any non-zero difference.

Exact vs. Asymptotic P-Values

For small samples (n < 20), exact p-values calculated from the theoretical null distribution provide more accurate inference than large-sample normal approximations. Most modern software offers both options.

Dealing with Ties

When ties are extensive, several adjustment methods exist. The Pratt method includes zero differences with their signs determined by a coin flip. The Wilcoxon-Pratt test can be more powerful in some situations but is less commonly implemented.

Power and Sample Size Planning

Power analysis for non-parametric tests is more complex than for parametric tests. General guidelines suggest the Wilcoxon signed-rank test has approximately 95% of the power of the paired t-test under normality, but can be much more powerful with skewed distributions. Use simulation-based approaches for precise power calculations.

Conclusion

The Wilcoxon signed-rank test remains an essential tool for modern data analysts working with paired samples that violate normality assumptions. By understanding its proper application, sizing samples with a power analysis, reporting effect sizes, and avoiding common pitfalls like post-hoc test selection and ignoring effect sizes, you can leverage this non-parametric technique to make robust, data-driven decisions.

Success with the Wilcoxon signed-rank test comes from three key practices: thorough pre-analysis planning including power calculations and clear hypothesis statements, rigorous verification of assumptions particularly regarding paired structure and symmetry of differences, and complete reporting that includes descriptive statistics, exact p-values, effect sizes, and confidence intervals contextualized with practical business implications.

As data analysis continues to evolve, the fundamental principles underlying the Wilcoxon signed-rank test (robustness to distributional assumptions, focus on median differences, and rank-based inference) ensure its continued relevance. Whether you're evaluating medical treatments, optimizing marketing campaigns, or testing product improvements, this time-tested method provides reliable insights that drive better business outcomes.

Writing this up for publication? See statistical analysis for researchers. Citations and R code included.

Frequently Asked Questions

What is the difference between the Wilcoxon signed-rank test and the paired t-test?

The Wilcoxon signed-rank test is a non-parametric alternative that does not assume normal distribution of differences, making it more robust when data is skewed or contains outliers. The paired t-test requires normally distributed differences and is more powerful when this assumption is met. If your differences are approximately normal, the paired t-test provides greater statistical power; if they're skewed or you have outliers you can't remove, the Wilcoxon test is more appropriate and reliable.

How do I interpret the p-value from a Wilcoxon signed-rank test?

A p-value less than your chosen significance level (typically 0.05) indicates statistically significant evidence that the median difference between paired observations is not zero. This suggests a real change or difference exists beyond random variation. However, always consider the p-value alongside effect size and confidence intervals: statistical significance doesn't automatically mean practical importance, especially with large samples.

What sample size do I need for the Wilcoxon signed-rank test?

The test can run with small samples, but with very few pairs it cannot reach small p-values at all (with 6 pairs, no zero differences and no ties, the smallest possible two-sided exact p-value is 0.031), and there is no single required number of pairs. The sample you need depends on the effect size you want to detect, the spread of the differences and the power you want (often 80%), so run a power analysis before collecting data.

Can I use the Wilcoxon signed-rank test with zero differences?

Zero differences (ties) are typically excluded from the analysis as they provide no information about direction of change. Most software automatically removes zeros and adjusts the sample size accordingly. If you have many zero differences (more than 20% of pairs), this may indicate measurement precision issues or that the sign test might be more appropriate than the signed-rank test.

When should I use Wilcoxon signed-rank test instead of other non-parametric tests?

Use this test specifically for paired or matched samples where you want to test whether the median difference is zero. For independent samples, use the Mann-Whitney U test instead. For more than two related groups, consider the Friedman test. If you can only determine the direction of change but not meaningful magnitudes, the sign test is more appropriate. The key distinguishing factor is having paired data with measurable differences.