Experiment & A/B Test Hypothesis Template

Use the [A/B Test Calculator](/tools). You need three inputs: your current baseline rate (e.g., 4.1% conversion), the minimum effect you want to detect (e.g., 12% relative lift), and your desired confidence level (95% is standard). The calculator outputs the required sample per variant. Divide that by your daily traffic to get the test duration. If the required duration is longer than 6 weeks, you need to either accept a larger MDE (detect only bigger effects) or find a higher-traffic test point.

It depends on your traffic and patience. For high-traffic pages (thousands of visitors per day), you can detect small effects (5-10% relative). For lower-traffic areas, you need to detect larger effects (15-25%) or the test will run too long. The business question is: "What is the smallest improvement that would change our decision?" If a 5% lift is not worth the engineering effort to maintain the treatment, set your MDE at 15% and design the test accordingly.

First, check if the test ran long enough. If you ended early (technical issue, executive pressure), extend it. If the test ran to full duration and the result is still inconclusive, you have learned something valuable: the change probably does not have a meaningful effect. Ship whichever variant is simpler to maintain. Document the result for the team so nobody re-runs the same test next quarter. See the [Product Analytics Handbook](/analytics-guide) for a framework on when to re-test vs. move on.

Yes, with caveats. If the experiments affect different elements (one tests the headline, another tests the CTA button), interaction effects are usually small and you can run them simultaneously. If the experiments affect related elements or the overall page layout, interaction effects can corrupt both results. In that case, run them sequentially. Some experimentation platforms (Optimizely, LaunchDarkly) support "mutual exclusion groups" that automatically prevent users from being in conflicting experiments.

Three options. First, accept a larger MDE (only detect effects of 20%+ relative). Second, run the test longer (8-12 weeks instead of 2-4). Third, use a different experiment type: fake door tests, concierge tests, or qualitative usability studies that require fewer participants. For [metrics like NPS](/metrics/net-promoter-score-nps) or satisfaction scores, you may need only 50-100 responses per variant to detect meaningful differences because the effect sizes are typically larger. ---