Purpose

Retention Rate: 100% (19/19 rows) - No observations were excluded during preprocessing, preserving the full experimental dataset
Rows Removed: 0 - No filtering, deduplication, or outlier removal occurred
Train/Test Split: Not applicable - This is descriptive analysis of completed experiments, not predictive modeling
Data Completeness: All 12 experiments retained despite 75% missing p-values, suggesting missing values were not treated as grounds for exclusion

This section identifies which experiments lack sufficient statistical power to draw reliable conclusions. Nine of the twelve experiments currently operate at 79% power—below the 80% threshold needed for confident decision-making. Understanding data sufficiency is critical because premature conclusions from underpowered tests risk false negatives, while extending tests unnecessarily delays business decisions.

Key Findings

Current Power Estimate: 0.79 across all insufficient experiments—just below the 80% target threshold, indicating borderline adequacy for statistical inference
Impressions Needed: Average of 3,747 additional impressions required (range: 574–6,518), representing roughly 2x the current collection in most cases
Timeline to Significance: Uniform 14-day extension needed across all nine experiments, suggesting consistent traffic patterns and effect sizes
Sample Size Variation: Current impressions range from 287 to 3,259, with smaller experiments (exp-027) requiring proportionally less additional data

Interpretation

The consistent 14-day recommendation across heterogeneous sample sizes indicates the power calculation accounts for both current impressions and expected traffic velocity. These experiments sit at the margin of statistical reliability; additional data collection will push them above the 80% power threshold, enabling defensible conclusions about treatment effects. The uniform timeline suggests traffic distribution is stable across experiment conditions.

Context

Power