LLM Fine-Tuning Plan Template for AI Products

For most tasks, 200-500 high-quality examples produce meaningful improvement. 1,000-2,000 examples typically reach diminishing returns for instruction-following tasks. The quality of examples matters far more than the count. 300 expert-curated examples will outperform 3,000 noisy auto-generated ones. Start with what you have, evaluate, and add more data only if the evaluation shows specific gaps.

Start with LoRA (Low-Rank Adaptation) for most projects. LoRA trains a small number of adapter weights instead of all model parameters, reducing compute cost by 60-90% and training time proportionally. Full fine-tuning is only necessary when LoRA results plateau and you need maximum quality. Most commercial use cases achieve their targets with LoRA.

Compare the fine-tuned model against your documented baselines on the held-out test set. If the fine-tuned model scores meaningfully higher on your primary metric and does not regress on general capabilities, it worked. If the improvement is marginal (less than 5% relative improvement), the cost of maintaining a custom model may not be justified. The [AI Eval Scorecard](/tools/ai-eval-scorecard) provides a structured approach for this comparison.

Overfitting means the model memorizes training examples instead of learning generalizable patterns. Signs include: training loss drops while validation loss increases, the model performs well on training examples but poorly on new inputs, or the model starts reproducing training examples verbatim. Prevent overfitting by using a validation set, training for fewer epochs, and ensuring your training data is diverse. The [hallucination glossary entry](/glossary/hallucination) covers how overfitting can manifest as confident but incorrect outputs.

Retrain when: your evaluation metrics degrade below acceptable thresholds, your product requirements change significantly, the base model provider releases a meaningful update, or you accumulate enough new training data to improve quality. For most products, quarterly retraining is sufficient. Set up automated evaluation pipelines that run weekly to detect when retraining is needed.