A/B Test Analysis Template for Product Analytics

Run the test until you reach your target sample size AND complete at least one full business cycle (typically 7 days for B2C, 14 days for B2B). Stopping early based on early significance inflates false positive rates. The [Product Analytics Handbook](/analytics-guide) covers sample size calculation in detail.

Statistical significance means the observed difference is unlikely to be due to chance. Practical significance means the difference is large enough to matter for your business. A 0.01% lift can be statistically significant with a large enough sample but is rarely worth the complexity of shipping. Always assess both.

Quantify the tradeoff. If conversion increases by 5% but page load time increases by 20%, calculate the net revenue impact of both changes. Document the tradeoff explicitly in the Decision section. If the guardrail degradation affects user trust or long-term retention, that typically outweighs short-term metric gains.

First, check whether segment-level results are adequately powered. Underpowered segments produce noisy estimates. If the segment differences are real and large, consider shipping the change only to the segments where it works (e.g., mobile only). Document the segment-specific decision rationale.

Pre-register your hypothesis, primary metric, and sample size before the test starts. Use this template to document those decisions in the Experiment Summary section before reviewing results. Run the Setup Validation checks. Do not add new metrics or segments after seeing the data. If you discover unexpected patterns, document them as hypotheses for future tests, not as conclusions from this one. The [AI ROI Calculator](/tools/ai-roi-calculator) can help quantify the expected value of ML-driven experimentation platforms. ---