# A/B testing: agent brief

> **Audience: an agent deciding whether to run this analysis on a dataset.** Not marketing
> copy. Terse, complete, and honest about failure. If you are asked "should I run an A/B
> test analysis here", you should be able to answer from this file alone, **including
> saying no**.
>
> Tool: `standard_ab_test` · free page: `/free/standard_ab_test` ·
> worked example: `/worked-examples/abtest` · lesson: *(pending operator watch)*

---

## 1. What it answers

**Did two versions of the same thing produce different outcomes, or is the gap chance?**

The distinguishing feature is not that two things are compared. It is that both versions
ran **at the same time, to comparable groups**, so the only systematic difference between
them is the thing you changed. A before/after comparison also compares two things and
answers nothing: the season, the campaigns and the traffic mix all moved too.

**Questions it is mistaken for:**

| Actually asked | Right tool |
|---|---|
| *Did the metric change after we shipped?* (no control group) | `standard_event_impact` |
| *Which of six variants is best?* | still this, but the multiplicity is real, see `standard_multiple_comparisons` |
| *How long until they churn?* (timing, censored) | `standard_survival` |
| *Where in the flow do people drop out?* | `standard_funnel` |
| *Is the average order value different?* (continuous outcome) | `standard_group_comparison` |
| *How many do I need for the NEXT test?* | power/sample size, not this. This reads a test that already ran |
| *What is our conversion rate?* | arithmetic. You do not need an analysis for one number |

## 2. When it applies, and when it does not

**Apply it when: assignment to versions was random or as-good-as-random, both ran over the
same period, and you have one row per subject.**

**Do not apply it when:**

- **There was no control.** "We shipped it and conversion rose" is a before/after, and every
  other thing that changed in that window is inside the number.
- **Assignment was not random.** Self-selected variants (opt-in beta, users who clicked a
  link) measure who chose, not what the change did.
- **You only have per-variant totals.** 126/900 and 140/1400 can be tested; "126 and 140"
  cannot, because the denominator is gone and no tool can recover it.
- **The outcome is continuous** (revenue, time on page, order value). This tool is built for
  a binary outcome per subject. Use `standard_group_comparison`.
- **The variants ran in different periods.** Then it is a before/after wearing an A/B label.

## 3. What the data must look like

`column_mapping` requires `variant` and `outcome`. Constraints: min 20 rows, max 100,000.

| key | type | meaning |
|---|---|---|
| `variant` | label | which version this subject saw |
| `outcome` | 0/1 | did the thing you care about happen |

**One row per subject, never pre-aggregated.** This is the constraint that bites hardest,
and it is not a preference: **the denominator only exists at row level.** A team that stored
"trials per page" has already thrown away how many people saw each page, and that number is
the entire analysis. Totals do not decompose.

**Unequal group sizes are fine and are not bias.** A ramped rollout produces them and the
random assignment still holds, so the rate comparison is valid. What unequal exposure
destroys is one particular *summary*, the raw count. See §5.

## 4. What it returns, and how to read each piece

| Output | Read it as | The trap |
|---|---|---|
| **Variant summary** | each variant's sample size and rate with a 95% interval | the sample sizes are the denominators. If they differ a lot, every count-based reading below is unsafe |
| **Outcome by variant** | the gap at a glance, bars with error bars | overlapping intervals usually mean the difference is not established. Usually, not always |
| **Absolute difference + 95% CI** | how many percentage points apart, and the plausible range | this is the number to report. A gap of 4 points with an interval of 1 to 7 is real **and possibly modest** |
| **Relative uplift** | the ratio between the rates | flattering and unstable on small baselines. A move from 0.2% to 0.3% is "+50%" and almost nothing |
| **p-value** | how often chance alone produces a gap this large | it says the gap is real, never how big. Report it beside the interval, never instead of it |
| **Exact-test fallback** | Fisher's, used when a cell gets small | if the tool discloses it fell back, the standard test was unreliable. That is a fact about your data, not a warning to dismiss |
| **AI insight** | plain-language verdict | reads the numbers it was given. It cannot know your assignment was broken |

## 5. How it fails

**The headline failure is that the tool answers exactly what you asked, and you asked with
the wrong denominator.** Counts and rates name opposite winners whenever exposure is
unequal, and both are arithmetically correct.

**Failure modes that yield a plausible wrong answer rather than an error:**

- **Ranking by raw conversions under a ramped rollout.** A page shown to more people
  accumulates more conversions however badly it performs. In the worked example the losing
  page wins the count 140 to 126 while converting at 10% against 14%. Nothing in the output
  flags this; the tool reports rates correctly and the *reader* uses counts.
- **Peeking.** Stopping when the p-value first dips below 0.05 inflates the false-positive
  rate far above 5%. The tool cannot see that you looked eleven times.
- **One-sided tests chosen after seeing the direction.** Honest only if the direction was
  fixed before the data existed.
- **Many variants or many metrics, uncorrected.** Each comparison gets its own chance to
  look real.
- **Statistical significance read as practical significance.** With enough traffic a
  meaningless gap becomes detectable.
- **Broken assignment.** Random assignment is an assumption this analysis cannot verify, and
  a violated one is invisible in the output.

## 6. Verified numbers you may cite

From `worked-example-abtest/VALIDATION.md`, agreed three ways (the notebook's R run, R's
`prop.test`/`chisq.test`/`fisher.test` called directly, and an independent scipy
reimplementation).

| quantity | value |
|---|---|
| control (existing page) | 126 trials / 900 visitors = 14.0000% |
| simplified (new page) | 140 trials / 1,400 visitors = 10.0000% |
| absolute difference | 4.0000 pp, favouring the **control** |
| 95% CI on the difference (Yates) | 1.1504 – 6.8496 pp |
| p (two-proportion / chi-square, Yates) | 0.0042274 |
| chi-square (1 df) | 8.18346 |
| p (Fisher exact, sensitivity) | 0.00402145 |
| relative uplift | 1.4000× |

**The count view crowns the loser:** simplified wins 140 to 126 on raw trials and loses
10.0% to 14.0% on rate, because the rollout ramped from a 50/50 split to roughly 30/70.
Every one of the four weeks has the control ahead on rate.

**Do not say** the unequal split invalidated the test. It did not. Assignment stayed random,
so the rate comparison holds; only the count summary is unusable.

**Do not say** these figures came from a platform run. See §8.

**Note the round numbers.** 14.0000% and 10.0000% are artifacts of a generator that fixed
counts rather than simulating. Real data does not land there. The teaching point is sound
and the arithmetic is checked three ways, but do not present the tidiness as evidence.

## 7. Where everything is

| | |
|---|---|
| Tool | `standard_ab_test` |
| Free page | https://mcpanalytics.ai/free/standard_ab_test |
| Worked example | *(rung 8, not yet published)* |
| Dataset | `/worked-examples/files/abtest_visitors.csv` |
| Notebook | `/worked-examples/files/abtest.html` · `abtest.Rmd` |
| Validation record | `lattice/v2/refs/LAT-2369-rmd-lesson-ladder/worked-example-abtest/VALIDATION.md` |
| Our tool's own run on this data | `rpt_R_qSrhTRclMilSR46Rx3gA` (job `mcp_standard_ab_test_2a150ed2e61b`, 2026-08-23, success) |

## 8. Open caveat, stated rather than hidden

The tool ran on this dataset and the job succeeded, but **the report was never opened and
its link has expired**, so the tool's output was never compared figure by figure against the
notebook. The reproduction is **attested, not re-checkable**. Every number in §6 rests on
the three-way local agreement. Until someone re-runs it and records the output values, do
not attribute these figures to a platform run.

## 9. Routing shortcut

```
Did both versions run at the same time, to comparable groups?
├── no → NOT this. A before/after is standard_event_impact, and self-selected groups
│        measure who chose, not what changed.
└── yes
    ├── one row per subject, with variant and a 0/1 outcome?
    │   ├── no, only per-variant totals → cannot run. The denominator is gone.
    │   └── yes
    │       ├── outcome continuous, not 0/1? → standard_group_comparison
    │       ├── more than two variants? → run it, then correct for multiplicity
    │       └── otherwise → run standard_ab_test
    └── report the RATE difference with its interval. If exposure was unequal, say so,
        and never rank the variants by raw conversions.
```