AI Incident Response Template for AI Products

A regular incident is binary: the system is working or it is not. An AI incident exists on a spectrum. The system returns 200 OK responses, but the outputs are wrong, biased, or harmful. This means standard uptime monitoring will not catch AI incidents. You need content-level monitoring, user feedback signals, and automated quality checks that traditional incident management does not include.

You need ML engineering expertise in the on-call rotation. AI incidents often require understanding model behavior, prompt engineering, and data pipeline analysis that backend engineers may not have. The most effective approach is a shared on-call with the ability to escalate to an ML specialist within 30 minutes for SEV-0 and SEV-1 incidents.

Run tabletop exercises quarterly. Create realistic scenarios: "The chatbot is telling users that our product cures allergies" or "The resume screener is scoring women's resumes 15% lower than men's." Walk through the detection, containment, investigation, and communication steps. These exercises reveal gaps in your runbook and build team confidence. See the [Responsible AI Framework](/frameworks/responsible-ai-framework) for a broader governance approach.

For SEV-0 and SEV-1: be transparent and fast. Acknowledge the issue, describe what you are doing to fix it, and explain what affected users should do. Avoid technical jargon. Do not say "the model hallucinated." Say "our AI provided incorrect information about [topic] and we have corrected it." For SEV-2 and SEV-3: no external communication is typically needed unless users report the issue publicly.

Every incident produces two outputs: a fix for the immediate issue and a preventive measure that addresses the root cause. The preventive measure should be a structural change (automated check, monitoring alert, test case, or process update) rather than a one-time fix. Track preventive actions to completion. If the same incident category recurs, the previous preventive measures were insufficient.