Canary Release Test Plan Template

Minimum bake time depends on your traffic volume. You need enough data to make a statistically meaningful comparison. For high-traffic products (>100K daily users), 4-6 hours per stage may suffice. For lower-traffic products (<10K daily users), each stage may need 24-48 hours to accumulate enough data. The key is having enough events in the canary group to detect a meaningful difference. If your conversion rate is 3% and your canary group sees 100 visitors, that is only 3 expected conversions. You cannot draw conclusions from that.

Start as small as practical. For backend changes, 1% is common. For frontend changes where you need visual feedback, 2-5% gives you enough user reports to catch issues. The goal of the first stage is to validate that the deployment works at all, not to measure business metrics. Business metric evaluation starts at 10-25% when the sample size is large enough. Track [error rates](/glossary/prioritization) at every stage.

They serve different purposes. A canary release validates that a change is safe to deploy (no errors, no performance degradation, no regressions). An A/B test measures whether a change improves a business metric. Use canaries for infrastructure changes, migrations, and bug fixes. Use A/B tests for feature experiments, UI changes, and pricing tests. For high-risk product changes, use both: canary first to validate safety, then A/B test to measure impact. The [bug report template](/templates/bug-report-template) helps track issues discovered during canary stages.

If you cannot tell whether metrics improved or degraded, the bake time was too short or the traffic allocation was too low. Extend the current stage. Do not advance to the next stage on ambiguous data. If after 3x the planned bake time the data is still inconclusive, the change likely has minimal impact. Decide whether to proceed based on qualitative factors and technical confidence.

Database migrations must be backward-compatible. The old code and new code will run simultaneously during the canary. The old code must work with the new schema, and the new code must work if a rollback reverts the deployment (but not the migration). Use expand-and-contract migration patterns: add new columns/tables first, deploy code that writes to both old and new, migrate reads, then remove old columns. The [e2e test template](/templates/e2e-test-template) can help validate that both code paths work correctly during the migration window. ---