Multivariate Test Template for Product Growth

Use A/B testing when you have one clear hypothesis to validate or when traffic is limited. Use MVT when you have 2-4 variables to optimize, enough traffic (typically 50K+ visitors/month to the test page), and you suspect variables may interact. A pricing page with a headline, CTA, and layout change is a good MVT candidate. A minor copy tweak is better as a simple A/B test. The [Product Analytics Handbook](/analytics-guide) covers how to choose between test types.

Roughly 8x more than a comparable A/B test for a 2x2x2 factorial design. If an A/B test needs 3,000 visitors per variant (6,000 total), the same MVT with 8 combinations needs about 24,000 visitors per combination (192,000 total). If your page gets 10K visitors/month, MVT is probably not practical. Stick to sequential A/B tests.

An interaction effect occurs when two changes together produce a result different from the sum of their individual effects. If headline change A lifts conversion 10% and CTA change B lifts it 8%, the expected combined lift is 18%. If the actual combined lift is 25%, there is a positive interaction (synergy). Interactions are the reason MVT exists. Without testing combinations, you would never discover them.

Full factorial tests every possible combination and is the standard choice for 2-3 variables with 2 variants each (4-8 combinations). Fractional factorial tests a strategically chosen subset and is necessary when the full factorial has too many combinations (e.g., 4 variables x 3 variants = 81 combinations). Fractional designs can detect main effects but may miss some interaction effects.

Apply a correction for multiple comparisons. The Bonferroni correction divides your significance threshold by the number of comparisons (e.g., p < 0.05/8 = p < 0.00625 for 8 combinations). This is conservative. The Benjamini-Hochberg procedure controls the false discovery rate and is less conservative. Either is acceptable. The key is to decide on your correction method before analyzing results. ---