Is the difference real? — Group differences analyses | MCP Analytics

Question

Is the difference real?

Compare two groups? Find what drives an outcome? Spot a trend? The question decides the statistics — get this wrong and everything downstream is wrong too.

How we actually do it

We pick the method from your data, not from a menu. Your file is profiled first — group count, distribution, variance — and the test follows from what's actually there.
Assumptions are tested and reported, not assumed. When one fails we switch method and say so in the write-up rather than quietly carrying on.
You get an effect size, not just a p-value. 'Significant' on 40,000 rows can still be too small to act on, so magnitude and its confidence interval are always shown.
Every number traces back to code. The R that produced each figure ships with the report, so anyone can re-run it.

See it on real data

Sequential A/B Test — Peek Safely — sample report Open full report →

That is the actual report this analysis produces — not a mock-up. Every analysis on this page delivers the same shape of output on your own data.

Group differences

Every analysis classified under this question.

Sequential A/B Test — Peek Safely

Can I stop this experiment early? Sequential testing with always-valid inference so peeking does not inflate false positives.

Rows split into groups

A/B Test Analysis

Did my variant win?

Rows split into groups

ANCOVA — Adjusted Group Comparison

Do my groups differ once I adjust for a covariate?

Rows split into groups

Categorical Association — Chi-Square & Fisher

Are these categories related?

One row per person or thing

Equivalence & Non-Inferiority (TOST)

Are these two things practically the SAME within a margin?

Rows split into groups

Event Impact — Did the Change Move the Metric?

Did the change move the metric?

One thing measured over time

Group Comparison — t-test, ANOVA & Nonparametric

Do my groups actually differ?

Rows split into groups

Multiple Comparisons Correction

Which results survive multiple-testing correction?

Rows split into groups

Paired Comparison — Before vs After

Did things change before vs after?

Before and after, same subject

Propensity Score Matching — Fairer Before/After Comparisons

Is the comparison fair once groups are matched?

Rows split into groups

Proportion Tests & Confidence Intervals

Did the conversion rate really differ?

Rows split into groups

Repeated-Measures ANOVA

Did the same subjects change across three or more conditions?

Rows split into groups

Other questions

Who are my natural groups?Segmentation Which way is this heading?Trends over time Where am I losing people?Growth & funnels What's actually driving this?Outcome drivers What is my survey telling me?Survey & ratings What does all the evidence say?Evidence synthesis What are the odds?Bayesian methods Did it actually cause it?Cause & effect Can I trust this data?Data quality Can I trust these predictions?Model quality

← Back to the full library