Prompt Testing Template for AI Products

For a production prompt, aim for 30-50 test cases minimum: 15-20 happy path, 5-10 edge cases, 5-10 adversarial, and 5-10 regression cases. High-risk features (financial, medical, legal) should have 100+ test cases. Start with what you can evaluate thoroughly. A small, well-scored test set is more valuable than a large one with inconsistent scoring.

Use both. Automated checks catch format compliance, required/prohibited content, and safety violations quickly and cheaply. Human evaluation catches nuanced quality issues (tone, helpfulness, coherence) that automated metrics miss. A good pattern is: automated checks as a gate (must pass), human scoring for quality dimensions. The [AI evaluation glossary entry](/glossary/ai-evaluation-evals) covers evaluation methodology in depth.

Run each test case 3-5 times at the same temperature setting. Score each run independently, then average the scores. If variance across runs is high for specific test cases, flag those as instability issues. Consider lowering temperature for features where consistency matters more than creativity. Report both the average score and the variance in your results.

Build automation when you are updating prompts more than once per month, when your test set exceeds 50 cases, or when multiple team members are editing prompts. Start manual and automate incrementally: first automate test execution (run all cases and collect outputs), then automate format/safety checks, then build a human scoring interface. The [prompt engineering glossary entry](/glossary/prompt-engineering) covers prompt lifecycle management.

Maintain a "golden set" of test cases with expected outputs that you re-run whenever the model version changes. Subscribe to the model provider's changelog and re-test within one week of any model update. If the provider offers model version pinning, use it for production and test new versions before switching. Document which model version each prompt was optimized for.