A/B Test Plan Template for Product Analytics

Run the test for the duration your power analysis calculated. For most mid-traffic SaaS pages (1,000-5,000 daily visitors), this is 2-4 weeks. Never run a test for less than one full business week (to account for day-of-week effects). If your sample size calculation says 21 days, run it for 21 days, even if the results look significant on day 7.

This is exactly why guardrail metrics exist. If the degradation is within your pre-defined acceptable range, ship the variant and monitor the guardrail closely post-launch. If it exceeds the threshold, do not ship. Investigate whether the degradation is a direct consequence of the change (causal) or a coincidence (correlation). When in doubt, revert and redesign.

Yes, but be aware that each additional variant increases the total sample size needed. A three-variant test (A/B/C) requires roughly 50% more traffic than a two-variant test to reach the same statistical power. For most teams, two variants (control + one change) is the right tradeoff between learning speed and statistical rigor. Reserve multi-variant tests for high-traffic surfaces where the extra duration cost is minimal.

Statistical significance tells you the result is unlikely to be due to chance. Practical significance tells you the result is large enough to matter for your business. A test can be statistically significant at p < 0.01 but show only a 0.1% lift in conversions, which might not justify the engineering cost to maintain the variant. Always evaluate both. Define your MDE based on what would be practically meaningful, not just what would be statistically detectable.

Test when the stakes are high and reversibility is low. Pricing pages, checkout flows, onboarding sequences, and core activation loops deserve formal A/B tests because a wrong decision has measurable revenue impact. For low-risk changes on low-traffic pages (a help article layout, a settings page tweak), just ship it and monitor metrics. The cost of running a formal test should be proportional to the risk of getting it wrong. ---