T-Test (Independent Samples): Compare Two Group Means

You have two groups and a number you care about. Did the new checkout flow actually increase order values, or did you just get lucky? Is there a real salary gap between departments, or is the difference within normal variation? The independent samples t-test gives you a statistically grounded answer. Upload a CSV and run the test.

Worked Example (Simulated): Did the New Checkout Flow Raise Order Value?

Question: Did the new checkout flow raise the mean order value?

Inputs: Made-up (simulated) data, seed 42. The script makes a control group and a new-flow group of 12 values each. It builds in an 8 point rise: control values are drawn around 50, new-flow values around 58, and both use a spread of 10. These are not real orders.

Method: Welch's two-sided t-test compares the group means. It checks how well the observed gap fits the null claim that the population means are equal. The script also prints the 95% confidence interval for the gap and the chance that a fresh draw of this size would reach p < 0.05, given the built-in gap.

Complete Python script:

import numpy as np
from scipy import stats

seed = 42
control_center, new_center, spread, n = 50, 58, 10, 12
rng = np.random.default_rng(seed)
control = rng.normal(loc=control_center, scale=spread, size=n)
new_flow = rng.normal(loc=new_center, scale=spread, size=n)
result = stats.ttest_ind(new_flow, control, equal_var=False)
mean_diff = new_flow.mean() - control.mean()
control_term = control.var(ddof=1) / len(control)
new_term = new_flow.var(ddof=1) / len(new_flow)
se = np.sqrt(control_term + new_term)
df = (control_term + new_term) ** 2 / (
    control_term ** 2 / (len(control) - 1)
    + new_term ** 2 / (len(new_flow) - 1)
)
critical = stats.t.ppf(0.975, df)
ci_low, ci_high = mean_diff - critical * se, mean_diff + critical * se

# Chance that a fresh draw reaches p < 0.05 when the built-in gap is 8
# (equal-variance t-test, same spread and group size).
built_in_gap = new_center - control_center
ncp = built_in_gap / (spread * np.sqrt(2 / n))
t_crit = stats.t.ppf(0.975, 2 * n - 2)
detect = 1 - stats.nct.cdf(t_crit, 2 * n - 2, ncp) + stats.nct.cdf(
    -t_crit, 2 * n - 2, ncp
)

print(f"Seed: {seed}")
print(
    f"Input settings: control center={control_center}, "
    f"new-flow center={new_center}, shared spread={spread}"
)
print(f"Built-in gap: {built_in_gap}")
print(f"Control sample size: {len(control)}")
print(f"New-flow sample size: {len(new_flow)}")
print(f"Control values: {np.round(control, 2).tolist()}")
print(f"New-flow values: {np.round(new_flow, 2).tolist()}")
print(f"Control mean: {control.mean():.2f}")
print(f"New-flow mean: {new_flow.mean():.2f}")
print(f"Mean difference (new flow minus control): {mean_diff:.2f}")
print(f"Welch t-statistic: {result.statistic:.3f}")
print(f"Welch degrees of freedom: {df:.2f}")
print(f"Two-sided p-value: {result.pvalue:.4f}")
print(f"95% confidence interval for mean difference: [{ci_low:.2f}, {ci_high:.2f}]")
print(
    f"Chance of p < 0.05 in a fresh draw (built-in gap {built_in_gap}): "
    f"{detect:.0%}"
)

Printed output:

Seed: 42
Input settings: control center=50, new-flow center=58, shared spread=10
Built-in gap: 8
Control sample size: 12
New-flow sample size: 12
Control values: [53.05, 39.6, 57.5, 59.41, 30.49, 36.98, 51.28, 46.84, 49.83, 41.47, 58.79, 57.78]
New-flow values: [58.66, 69.27, 62.68, 49.41, 61.69, 48.41, 66.78, 57.5, 56.15, 51.19, 70.23, 56.45]
Control mean: 48.58
New-flow mean: 59.04
Mean difference (new flow minus control): 10.45
Welch t-statistic: 2.996
Welch degrees of freedom: 20.62
Two-sided p-value: 0.0070
95% confidence interval for mean difference: [3.19, 17.71]
Chance of p < 0.05 in a fresh draw (built-in gap 8): 47%

Result: The control mean is 48.58 and the new-flow mean is 59.04. The gap, new flow minus control, is 10.45. The t-statistic is 2.996 and the p-value is 0.0070. The 95% confidence interval for the gap is 3.19 to 17.71, which contains the built-in gap of 8. This draw shows a gap larger than the 8 points built in. With 12 per group, the script puts the chance of p < 0.05 for a gap of 8 at 47%, so a rerun with another seed can give a smaller gap and a larger p-value. In this sample the result is evidence of a higher mean for the new flow. It does not prove the flow caused the gap.

Limit: These values are simulated from a simple draw. They do not show what real users would do. A missed built-in effect would mean low power, not no effect.

Sources

• Welch, B. (1947). The generalization of "Student's" problem when several different population variances are involved. Biometrika. doi 10.1093/biomet/34.1-2.28. Welch published the unequal-variance t-test in 1947 (Biometrika).

What Is a T-Test?

The independent samples t-test compares the means of two groups. It checks how well the observed gap between two group means fits the null claim that the population means are equal. A small p-value is evidence against that claim, but it does not prove a real effect.

Here is the intuition. See the simulated worked example above. A gap between two group averages could reflect a real difference, or it could come from normal customer-to-customer variation. The t-test calculates how unlikely a gap that large would be if the population means were equal. If that probability falls below the threshold you chose in advance (commonly 5%), you treat the difference as statistically significant.

Mathematically, the t-test computes a t-statistic: the difference between the two group means divided by the standard error of that difference. The standard error accounts for both the sample sizes and how much variation exists within each group. A large t-statistic means the gap between groups is large relative to the noise, and the corresponding p-value tells you how likely a gap that large (or larger) would be if there were truly no difference between the groups.

When to Use the T-Test

Use the t-test whenever you have exactly two groups and a numeric outcome you want to compare. The classic scenarios span every domain:

A/B testing. You split website traffic between two variants and measure conversion rate, revenue per visitor, or time on page. The t-test tells you whether the winning variant genuinely outperforms the loser, or whether the observed difference is within the range of normal fluctuation. This is the foundation of data-driven product decisions at companies of every size.

Clinical studies. A treatment group receives a drug and a control group receives a placebo. The t-test determines whether the treatment group's outcomes (blood pressure reduction, symptom improvement, recovery time) are significantly better than the control group's.

HR and compensation analysis. Compare average salaries between two departments, genders, or office locations. The t-test quantifies whether an observed pay gap is statistically significant or falls within normal variation given sample sizes and salary dispersion. This is critical for pay equity audits and compliance reporting.

Marketing campaigns. You ran Campaign A (email-only) and Campaign B (email plus retargeting). Compare the response rate, average order value, or customer acquisition cost between the two campaigns. The t-test prevents you from declaring a winner based on a difference that could easily be noise.

The key constraint is exactly two groups. If you have three or more groups (e.g., four ad variants, five sales regions), use ANOVA instead. Running multiple t-tests across many pairs inflates your false positive rate, ANOVA handles this correctly in a single test.

What Data Do You Need?

You need a CSV file with at least two columns:

group_var: a categorical column that identifies which group each observation belongs to. This column must have exactly two unique values. Examples: "treatment" and "control", "Version A" and "Version B", "Male" and "Female", "Before" and "After". If your column has three or more unique values, the tool will flag it and you should use ANOVA instead.

measurement: a numeric column containing the outcome you want to compare. Examples: order value in dollars, test score, response time in seconds, satisfaction rating on a 1-10 scale. This must be a continuous (or at least interval-scale) numeric variable, not a category.

The tool requires a minimum of 5 per group to run. With small groups the test has limited statistical power, meaning it may fail to detect real differences, and the normality assumption becomes more important, so the report's normality diagnostics deserve close attention. How many observations you need depends on the size of the effect you expect and on the spread of the data.

The tool uses Welch's t-test by default, which does not assume equal variances between the two groups. This is the safer and more modern approach. If you know your groups have equal variances (or want to test it), the module_parameters include a var_equal option that switches to the classic Student's t-test. In practice, Welch's test performs as well or better than Student's in nearly all cases, so the default is almost always the right choice. The report includes a variance equality test (F-test) so you can verify this for yourself.

Additional module_parameters let you set the significance_level (default 0.05), confidence_level (default 0.95), and alternative hypothesis direction ("two.sided", "less", or "greater"). For most analyses, the defaults are appropriate.

How to Read the Report

Each part of the report serves a specific purpose. Here is what to look for in each one.

Analysis Overview. This section sets the stage: which groups are being compared, how many observations are in each, and what measurement is being tested. Verify that the groups and sample sizes match your expectations. If you see unexpected group labels, your group column may contain data quality issues (typos, extra whitespace, mixed casing).

Data Preprocessing. Shows what the tool did to prepare your data before running the test: handling missing values, identifying and flagging outliers, verifying the two-group constraint. If rows were removed, this section explains why. Check this if your sample sizes are smaller than expected.

Executive Summary. The plain-English bottom line. This section tells you whether the difference between groups is statistically significant, how large it is in practical terms, and what it means for your decision. If you only read one section, read this one. It synthesizes the t-statistic, p-value, and effect size into an actionable conclusion.

Distribution Comparison. Overlapping histograms or density plots showing how the two groups' values are distributed. This is your first visual check. If the two distributions overlap heavily, the groups are similar. If they are clearly separated, the groups probably differ, and the p-value and effect size tell you whether that difference is significant and meaningful. Look for skewness, bimodality, or outliers that might affect the test's reliability.

Box Plot Comparison. Side-by-side box plots showing the median, interquartile range (25th to 75th percentile), and outliers for each group. Box plots make it easy to compare central tendency and spread at a glance. Box plots give a visual check of the group medians and spread. Use the test result and confidence interval to judge the evidence for a difference. Outliers (individual dots beyond the whiskers) can pull the mean away from the median, if you see many outliers, the Mann-Whitney test may be more appropriate.

Normality Diagnostics (QQ Plot). Quantile-quantile plots and Shapiro-Wilk test results for each group. The QQ plot charts each group's data against what you would expect from a perfect normal distribution. Points falling along the diagonal line indicate normality. Deviations at the tails (S-curves or hooks) indicate skewness or heavy tails. The Shapiro-Wilk p-value below 0.05 formally rejects normality. For larger samples, the t-test is fairly robust to moderate non-normality thanks to the Central Limit Theorem. For small samples with clear normality violations, consider the Mann-Whitney U test.

Effect Size. Cohen's d quantifies how large the difference is in standardized units (standard deviations). This is arguably more important than the p-value because it tells you whether the difference matters in practice, not just whether it is statistically detectable. The standard benchmarks: d = 0.2 is a small effect (the difference exists but is subtle), d = 0.5 is a medium effect (noticeable and potentially meaningful), and d = 0.8 or higher is a large effect (the groups are clearly different). A statistically significant result (p < 0.05) with a tiny effect size (d < 0.2) means the difference is real but probably too small to act on. Conversely, a non-significant result with a medium or large effect size suggests you may need more data.

Test Results. The core statistical output. The t-statistic tells you how many standard errors apart the two group means are. The degrees of freedom (adjusted for Welch's correction) indicate the effective sample size. The p-value is calculated under the null claim that the population means are equal. It is the chance, under that claim, of a t-statistic at least as far from zero as the observed one. A p-value below your significance level (typically 0.05) means you reject the null hypothesis and conclude the groups differ. The confidence interval for the mean difference shows the range of plausible values for the true difference, if it does not include zero, the difference is significant at your chosen confidence level.

Summary Statistics. Group-level descriptive statistics: mean, standard deviation, median, minimum, maximum, and sample size for each group. Use this to ground the abstract test results in concrete numbers. The mean difference and its direction are immediately visible here. If the standard deviations are very different between groups (e.g., one is three times the other), Welch's correction is doing important work. The classic Student's t-test would be unreliable.

When to Use Something Else

The t-test is the right tool for two-group, numeric-outcome comparisons when the data is approximately normal. But several common situations call for a different approach.

If your data is heavily skewed, has extreme outliers, or is measured on an ordinal scale (like a 1-5 Likert rating), use the Mann-Whitney U test. It compares ranks rather than means, so it does not assume the data are normally distributed. The t-test report's normality diagnostics will flag when this is the better choice. Try it free.

If you have three or more groups, use ANOVA. Running separate t-tests for every pair of groups inflates your false positive rate. With four groups, that is six pairwise comparisons and roughly a 26% chance of at least one false positive (1 - 0.95^6, treating the tests as independent). ANOVA handles all groups in a single test with proper error control, and its post-hoc tests (Tukey HSD) identify which specific pairs differ. Try it free.

If the same subjects are measured twice (before and after an intervention, or under two conditions), you need a paired t-test, not the independent samples version. The paired test accounts for within-subject correlation and is substantially more powerful in this design. The independent samples t-test treats each observation as coming from a different person, which wastes information and reduces sensitivity when the data is actually paired.

If your outcome variable is categorical rather than numeric (for example, comparing pass/fail rates or yes/no responses between two groups) use a chi-square test. The t-test requires a numeric measurement; it cannot analyze proportions or counts in a contingency table. Try it free.

If your data has more than two groups and is non-normal, the Kruskal-Wallis test is the non-parametric analogue of ANOVA. It handles three or more groups without assuming normality. Try it free.

The R Code Behind the Analysis

Every report names its method and includes the R code that produced it, so you can reproduce, audit and cite it. The method code is fixed, not rewritten for each run, so the same data gives the same analysis every time.

The analysis uses t.test() from base R with Welch's correction enabled by default (var.equal = FALSE). This is the same function used in university statistics courses, textbooks, and peer-reviewed research. The effect size is computed using Cohen's d from the effsize package. Normality is assessed with shapiro.test() (Shapiro-Wilk test) and QQ plots, while variance equality is checked using var.test() (F-test for ratio of variances). If you set var_equal = TRUE in the module parameters, the analysis switches to the classic Student's t-test (var.equal = TRUE), but the default Welch variant is recommended for almost all real-world data. Every step is visible in the code tab of your report, so you or a statistician can verify exactly what was done.