Chaos Engineering Experiment Template

Frame it in terms of incident prevention and cost. Every chaos experiment that finds a gap before production is an incident that did not happen. Incidents cost engineering time, customer trust, and potentially revenue. Start with low-risk experiments in staging to build confidence, then present the findings (gaps discovered, fixes applied) to justify expanding to production.

For Kubernetes: Chaos Mesh, LitmusChaos, or Gremlin. For AWS: AWS Fault Injection Simulator (FIS). For general-purpose: Gremlin (commercial, multi-platform). For simple experiments: a bash script that kills processes or blocks network ports is a valid starting point. The tool matters less than the process. This template works regardless of tooling.

Run experiments monthly on critical services and quarterly on supporting services. After any major architecture change, run the relevant experiments again to re-validate. Game days (larger-scale chaos exercises involving multiple teams) should happen quarterly. The goal is to make resilience testing routine, not a one-time event.

This is why abort conditions and blast radius controls exist. If an experiment causes impact beyond the defined scope, activate the abort procedure immediately, then treat it as a real incident with a post-mortem. The post-mortem should cover both the system failure and the experiment's safety controls. Tighten the blast radius for future experiments.

PMs should understand the reliability posture of their product and the business impact of potential failures. They do not need to attend experiments, but they should review the findings. If a chaos experiment reveals that database failover causes 4 minutes of checkout downtime, the PM needs to know that and factor it into reliability investment decisions.