ANOVA: Compare Means Across Groups

You have three or more groups and a number you care about: revenue, satisfaction score, conversion rate, test results. Are the differences between groups real, or just noise? ANOVA gives you a straight answer backed by an F-test, post-hoc comparisons, and effect sizes. Upload a CSV and run the analysis.

Worked Example: Petal Length by Iris Species

Question: Do mean petal lengths differ across the three Iris species?

Data: the Iris data set shipped with scikit-learn: 150 flowers, 50 per species, petal length in cm. It is public, real data. It is not simulated and not customer data.

Method: One-way ANOVA tests whether group means differ. Tukey HSD checks each pair and adjusts for all the pair tests.

Result: Mean petal length differed by species: 1.462 cm, 4.260 cm and 5.552 cm. The F test gave F = 1180.161 with p below 0.001, and eta-squared was 0.941. Each pair differed after Tukey HSD (p below 0.001 for all three). This data cannot show what caused the gaps.

Limit: Nothing is built in here, because the data are real. The three species have spreads that differ about three-fold (SD 0.17, 0.47 and 0.55 cm). One-way ANOVA and Tukey HSD assume similar spread. The gaps are so large that the answer holds, but the F value and the tiny p should not be read as exact when spreads are unequal. In 1951 Welch published a test for comparing several means that does not need equal spread.

Sources

The Iris data were published by R. A. Fisher in 1936 (Annals of Eugenics). Links for the example:

• scikit-learn, load_iris (Iris data)
• SciPy, f_oneway (one-way ANOVA)
• SciPy, tukey_hsd (Tukey HSD)
• SciPy, kruskal (Kruskal-Wallis test)
• NIST e-Handbook, 7.4.1 The Kruskal-Wallis test
• Penn State STAT 501, Lesson 13: Weighted Least Squares & Logistic Regressions
• R manual, aov (Fit an Analysis of Variance Model)
• R manual, TukeyHSD (Compute Tukey Honest Significant Differences)

Script and printed output

from sklearn.datasets import load_iris
from scipy import stats
import numpy as np

data = load_iris()
col = data.feature_names.index('petal length (cm)')
y = data.data[:, col]
group_ids = data.target
names = data.target_names
groups = [y[group_ids == i] for i in range(len(names))]


def fmt_p(p):
    return 'below 0.001' if p < 0.001 else f'{p:.3f}'


print('Dataset: Iris data shipped with scikit-learn')
print('Question: Do mean petal lengths differ by species?')
for name, values in zip(names, groups):
    print(f'{name}: n={len(values)}, mean={np.mean(values):.3f} cm, '
          f'SD={np.std(values, ddof=1):.3f} cm')
result = stats.f_oneway(*groups)
mean_all = np.mean(y)
ss_between = sum(len(g) * (np.mean(g) - mean_all) ** 2 for g in groups)
ss_total = sum((v - mean_all) ** 2 for v in y)
print(f'One-way ANOVA: F={result.statistic:.3f}, p {fmt_p(result.pvalue)}')
print(f'Eta-squared: {ss_between / ss_total:.3f}')
tukey = stats.tukey_hsd(*groups)
ci = tukey.confidence_interval()
print('Tukey HSD pairwise differences '
      '(first group minus second; 95% CI; adjusted p):')
for i in range(len(names)):
    for j in range(i + 1, len(names)):
        print(f'{names[i]} - {names[j]}: {tukey.statistic[i, j]:.3f} cm; '
              f'[{ci.low[i, j]:.3f}, {ci.high[i, j]:.3f}]; '
              f'p {fmt_p(tukey.pvalue[i, j])}')

Printed output of that script:

Dataset: Iris data shipped with scikit-learn
Question: Do mean petal lengths differ by species?
setosa: n=50, mean=1.462 cm, SD=0.174 cm
versicolor: n=50, mean=4.260 cm, SD=0.470 cm
virginica: n=50, mean=5.552 cm, SD=0.552 cm
One-way ANOVA: F=1180.161, p below 0.001
Eta-squared: 0.941
Tukey HSD pairwise differences (first group minus second; 95% CI; adjusted p):
setosa - versicolor: -2.798 cm; [-3.002, -2.594]; p below 0.001
setosa - virginica: -4.090 cm; [-4.294, -3.886]; p below 0.001
versicolor - virginica: -1.292 cm; [-1.496, -1.088]; p below 0.001

What Is ANOVA?

ANOVA, Analysis of Variance, answers one question: are the averages across my groups meaningfully different, or could the differences I see be explained by random variation? Imagine you run three sales regions and want to know if the West region genuinely outperforms the others, or if it just had a lucky quarter. ANOVA tests all three groups simultaneously and tells you whether at least one group stands apart from the rest.

The intuition is straightforward. ANOVA compares how much the group averages spread out from each other (between-group variance) against how much individual data points scatter within each group (within-group variance). If the groups are far apart relative to the noise inside each group, the F-statistic is large and the p-value is small, meaning the differences are unlikely to be random chance.

For example, suppose you are comparing average order value across three marketing channels: paid search, social, and email. Each channel has dozens of orders with natural variation. ANOVA looks at the gap between the channel averages and weighs it against the typical order-to-order variation within each channel. A significant result means at least one channel is pulling different customers (or different buying behavior) compared to the others.

When to Use ANOVA

The most common business use case is comparing performance across three or more segments. You might compare conversion rates across four landing page variants (an A/B/C/D test), revenue per customer across five product tiers, or satisfaction scores across regional offices. Anytime you have a numeric outcome and a categorical grouping variable with three or more levels, one-way ANOVA may fit, if its rules hold.

In research settings, ANOVA compares group means, such as test scores across three teaching methods, patient outcomes across four drug dosages, or crop yields across different fertilizer blends.

A critical point: you cannot simply run multiple t-tests to compare every pair of groups. If you have four groups, that is six pairwise comparisons. If each has a 5% chance of a false positive, and no test affects another, the chance of one or more false positives is about 26%. Pair tests may affect each other. Tukey HSD adjusts for the whole set of tests. ANOVA handles all groups in a single test, keeping your error rate at the level you set. When ANOVA finds a significant difference, the Tukey HSD post-hoc test then tells you which specific pairs of groups differ, with proper correction for multiple comparisons.

What Data Do You Need?

You need a CSV with at least two columns: one categorical column that defines your groups (like region, campaign, treatment, or product tier) and one numeric column that holds the measurement you want to compare (like revenue, score, duration, or count). The tool will ask you to map which column is which when you upload.

For reliable results, aim for at least three groups with five or more observations in each. More observations per group means more statistical power. That is, a better chance of detecting real differences if they exist. Very small groups (under five) can still produce results, but the test loses sensitivity and the assumptions become harder to verify.

ANOVA assumes that the data within each group is approximately normally distributed and that the variance is roughly similar across groups. In practice, ANOVA is robust to mild violations of both assumptions, especially with larger samples. The report includes assumption checks, Levene's test for equal variances and normality diagnostics, so you will know if your data is a good fit. If the assumptions are badly violated, the report will flag it and suggest alternatives like the Kruskal-Wallis test.

Worked Example (synthetic data)

This example uses synthetic numbers made up for illustration. They are not customer data. Suppose three groups of five orders each, and a satisfaction score for each:

Grand mean            = 27.0
Between-group SS      = 5 x (23-27)^2 + 5 x (28-27)^2 + 5 x (30-27)^2 = 80 + 5 + 45 = 130
Within-group SS       = 10 + 10 + 10 = 30     (each group: sum of squared gaps from its own mean)
Degrees of freedom    = 3 - 1 = 2 between, 15 - 3 = 12 within
Mean squares          = 130 / 2 = 65.0 between, 30 / 12 = 2.5 within
F statistic           = 65.0 / 2.5 = 26.0 on (2, 12) degrees of freedom
p-value               = 4.3e-5
Eta-squared           = 130 / (130 + 30) = 0.81
Tukey HSD (95%)       = 3.773 x sqrt(2.5 / 5) = 2.67

Reading it: the F-test says the three means are not all equal, and about 81 percent of the variation in these scores sits between groups. The F-test does not say which groups differ. Tukey's comparison does: A and B differ by 5.0, A and C by 7.0, both more than 2.67, so both are significant. B and C differ by 2.0, less than 2.67, so that pair is not significant. All three groups have the same variance (2.5) in this made-up data, so the equal-variance assumption holds by construction. Real groups rarely line up this neatly, which is why the report runs Levene's test and normality diagnostics first.

How to Read the Report

The report starts with the ANOVA table, which shows the F-statistic and p-value. The F-statistic is a ratio: between-group variance divided by within-group variance. A larger F means the groups are more different from each other relative to the noise within groups. The p-value tells you the probability of seeing an F-statistic this large if all groups were actually identical. A p-value below 0.05 is the conventional threshold for concluding that at least one group differs from the rest.

Next comes the Tukey HSD (Honestly Significant Difference) table. ANOVA only tells you that some difference exists, Tukey tells you where. It lists every pairwise comparison (Group A vs. Group B, Group A vs. Group C, and so on) with a confidence interval and adjusted p-value for each pair. Pairs with p-values below 0.05 have a statistically significant difference. The confidence intervals show the estimated size and direction of the difference, which is often more useful than the p-value alone.

The box plots give you a visual comparison of the distributions. Each box shows the median, interquartile range, and outliers for a group. Even before you look at the numbers, the box plots often make the story obvious, overlapping boxes suggest similar groups, while separated boxes suggest real differences. The report also includes effect size (eta-squared), which measures practical significance. A statistically significant result with a tiny effect size means the difference is real but possibly too small to matter for your decision.

When to Use Something Else

If you only have two groups, use a t-test instead. ANOVA with two groups produces mathematically identical results to a t-test, but the t-test output is simpler to interpret and gives you a directional result (which group is higher). Save ANOVA for three or more groups.

If your data is heavily skewed, has extreme outliers, or is measured on an ordinal scale (like a 1-5 rating), consider the Kruskal-Wallis test. It is the non-parametric alternative to ANOVA. Kruskal-Wallis compares ranks, or the order of values, across groups. Its result can test medians when group shapes are alike. The ANOVA report flags when your data violates normality assumptions, so you will know when to switch.

If you want to compare groups while accounting for a confounding variable (for example, comparing sales regions while controlling for store size) you need ANCOVA (Analysis of Covariance). And if your outcome variable is categorical rather than numeric (for example, comparing pass/fail rates across groups), use a chi-square test instead.