How many labeled examples do we need?

It depends on the task complexity and the number of categories. For text classification with 5-10 categories, 1,000-5,000 labeled examples per category is a reasonable starting point. For fine-tuning LLMs, even 100-500 high-quality examples can yield meaningful improvements with techniques like LoRA.

Should we label in-house or use a vendor?

In-house labeling produces higher quality for domain-specific tasks but is expensive and slow. Vendors are faster and cheaper but require thorough guidelines and heavy QC. Most teams use a hybrid: in-house experts create guidelines and gold standards, vendors handle production volume.

What inter-rater reliability score is good enough?

A Cohen's kappa of 0.80 or higher is generally considered "substantial agreement" and is sufficient for most ML tasks. For high-stakes applications (medical, legal, financial), aim for 0.85+. Below 0.70, your guidelines likely need revision.

How do we handle labeler disagreements?

Disagreements are data, not failures. Track disagreement rates by label category to identify where your taxonomy is ambiguous. Have a lead annotator make the final call, and add the disputed example to your edge cases documentation.

Can we use LLMs to replace human labeling entirely?

Not yet for most production use cases. LLM-generated labels work well for pre-annotation (reducing human effort by 30-50%) and for prototyping when you need quick-and-dirty training data. But for production models, human-verified labels still produce more reliable training data, especially for domain-specific or nuanced categories.

AI Data Labeling Template for AI Products

What This Template Is For

The quality of your AI model is bounded by the quality of your training data, and the quality of your training data is bounded by the quality of your labeling process. Poorly defined labeling guidelines produce inconsistent annotations. Inconsistent annotations produce noisy training data. Noisy training data produces a model that fails in unpredictable ways. This chain of failure is one of the most common reasons AI features underperform, and it is almost always preventable with a structured labeling plan.

This template helps product and data teams plan every aspect of a data labeling workflow: defining the labeling taxonomy, writing clear annotator guidelines, setting up quality control processes, measuring inter-rater reliability, and managing the feedback loop between labeling results and model performance. For teams in the early stages of defining their AI data needs, the AI Data Requirements Template covers upstream data sourcing and schema decisions. The AI PM Handbook includes a full chapter on data strategy for AI products.

Good labeling is expensive and time-consuming. This template helps you spend that budget effectively by making labeling decisions explicit before work begins. It covers both human labeling (in-house teams or vendors like Scale, Labelbox, or Appen) and semi-automated labeling with LLM-assisted pre-annotation. If your project involves evaluating model outputs rather than labeling inputs, the AI Model Evaluation Template is the better fit. For teams building RAG systems, labeling relevance judgments for retrieval quality is a common use case this template supports.

When to Use This Template

When starting a new ML project that requires labeled data. Define your labeling plan before you start collecting annotations.
When labeling quality is inconsistent. Use the quality control section to diagnose and fix reliability issues.
When scaling labeling from a small team to vendors. Document guidelines thoroughly enough that new annotators can ramp up quickly.
When transitioning from manual to semi-automated labeling. Plan the human-in-the-loop workflow for LLM-assisted pre-annotation.
When debugging model performance issues. Poor labeling is a common root cause. Audit your annotations before blaming the model.
When onboarding a new labeling vendor. Share this document as the spec for what you need and how quality will be measured.

How to Use This Template

Define your labeling taxonomy. List every label, category, or annotation type. Include clear definitions and boundary cases for each.
Write annotator guidelines. For each label, provide a definition, 2-3 positive examples, 2-3 negative examples, and rules for edge cases. Ambiguous guidelines are the top source of labeling errors.
Set up quality control. Define your gold standard set, inter-rater reliability target, and review sampling rate.
Plan annotator management. Document who is labeling, how they are trained, how performance is measured, and how disputes are resolved.
Establish the feedback loop. Connect labeling metrics to model performance metrics so you can trace model failures back to labeling issues.

The Template

## Data Labeling Plan

**Project Name**: [Name]
**Date**: [YYYY-MM-DD]
**Owner**: [PM or Data Lead]
**Model Type**: [Classification / NER / Segmentation / Ranking / Generation eval]
**Target Dataset Size**: [Number of labeled examples]

---

### 1. Labeling Taxonomy

| Label / Category | Definition | Example (Positive) | Example (Negative) |
|-----------------|-----------|-------------------|-------------------|
| [Label A] | [Clear 1-sentence definition] | [Example that should get this label] | [Example that should NOT get this label] |
| [Label B] | [Clear 1-sentence definition] | [Example] | [Counter-example] |
| [Label C] | [Clear 1-sentence definition] | [Example] | [Counter-example] |

**Multi-label allowed?** [Yes / No]
**"Uncertain" or "Skip" label allowed?** [Yes / No. If yes, when to use it]

---

### 2. Annotator Guidelines

#### General Rules
- [Rule 1: e.g., "When in doubt, choose the more specific label"]
- [Rule 2: e.g., "If the text is ambiguous, label based on the most likely interpretation"]
- [Rule 3: e.g., "Do not use external knowledge. Label only based on the provided text"]

#### Edge Cases
| Scenario | Correct Label | Rationale |
|----------|-------------|-----------|
| [Ambiguous case 1] | [Label] | [Why this label is correct] |
| [Ambiguous case 2] | [Label] | [Why] |
| [Ambiguous case 3] | [Label] | [Why] |

#### Annotation Format
- **Input format**: [Text / Image / Audio / Video / Structured data]
- **Output format**: [Single label / Multi-label / Bounding box / Span annotation / Rating scale]
- **Tool**: [Labelbox / Scale / Prodigy / Label Studio / Custom]

---

### 3. Quality Control

| QC Mechanism | Details |
|-------------|---------|
| Gold standard set size | [N examples with expert labels] |
| Gold standard check frequency | [Every N items] |
| Minimum accuracy on gold set | [e.g., 90%] |
| Inter-rater reliability target | [e.g., Cohen's kappa >= 0.80] |
| Review sampling rate | [e.g., 10% of all annotations reviewed by lead] |
| Dispute resolution process | [Who decides when annotators disagree?] |

#### Reliability Measurement
- **Metric**: [Cohen's kappa / Fleiss' kappa / Krippendorff's alpha / % agreement]
- **Measurement frequency**: [Every N labeled items]
- **Action if below threshold**: [Retrain annotators / Revise guidelines / Escalate]

---

### 4. Annotator Management

| Role | Count | Source | Training |
|------|-------|--------|----------|
| Lead annotator | [N] | [In-house / Vendor] | [Expert, defines gold standard] |
| Annotator | [N] | [In-house / Vendor] | [Guideline review + 50-item calibration] |
| Reviewer | [N] | [In-house] | [Resolves disputes, audits quality] |

**Onboarding process**: [Steps for new annotators]
**Performance tracking**: [Accuracy, speed, consistency metrics per annotator]
**Removal criteria**: [When to reassign or remove underperforming annotators]

---

### 5. Semi-Automated Labeling (Optional)

| Step | Details |
|------|---------|
| Pre-annotation model | [LLM / existing classifier / rule-based] |
| Pre-annotation accuracy estimate | [%] |
| Human review rate | [% of pre-annotations reviewed by humans] |
| Correction tracking | [How corrections feed back to improve pre-annotation] |

---

### 6. Timeline and Budget

| Phase | Duration | Volume | Cost |
|-------|----------|--------|------|
| Guideline development | [N days] | N/A | [Internal hours] |
| Pilot labeling (calibration) | [N days] | [N items] | $[Amount] |
| Production labeling | [N weeks] | [N items] | $[Amount] |
| Quality audit | [N days] | [N items reviewed] | [Internal hours] |
| **Total** | | **[N items]** | **$[Total]** |

---

### 7. Feedback Loop

- [ ] Labeling metrics are tracked alongside model performance metrics
- [ ] Model error analysis includes labeling quality check
- [ ] Edge cases found during model eval are added to annotator guidelines
- [ ] Gold standard set is updated quarterly with new edge cases

Filled Example: Customer Intent Classification

## Data Labeling Plan

**Project Name**: SupportBot Intent Classifier v2
**Date**: 2026-02-20
**Owner**: Priya Sharma, Data PM
**Model Type**: Multi-class text classification
**Target Dataset Size**: 25,000 labeled support tickets

---

### 1. Labeling Taxonomy

| Label / Category | Definition | Example (Positive) | Example (Negative) |
|-----------------|-----------|-------------------|-------------------|
| billing_issue | Customer has a problem with charges, invoices, or payment methods | "I was charged twice for my subscription" | "How much does the Pro plan cost?" (this is pricing_question) |
| feature_request | Customer asks for a new capability or improvement | "It would be great if I could export to PDF" | "The export button is broken" (this is bug_report) |
| bug_report | Customer reports something that is not working as expected | "The dashboard crashes when I filter by date" | "I wish the dashboard loaded faster" (this is feature_request) |
| account_access | Customer cannot log in or access their account | "I forgot my password and the reset link expired" | "Can I add another user to my account?" (this is account_management) |
| account_management | Customer wants to change account settings, users, or permissions | "How do I upgrade from Basic to Pro?" | "I cannot log in" (this is account_access) |
| pricing_question | Customer asks about pricing, plans, or discounts | "Do you offer annual billing?" | "I was charged the wrong amount" (this is billing_issue) |
| cancellation | Customer wants to cancel or downgrade their subscription | "I want to cancel my account" | "I want to switch to a cheaper plan" (this is account_management) |
| other | Does not fit any of the above categories | "What are your office hours?" | |

**Multi-label allowed?** No. Choose the primary intent.
**"Uncertain" or "Skip" label allowed?** Yes. Use "uncertain" when the ticket is genuinely ambiguous after reading the full text. Target: fewer than 3% of labels.

---

### 2. Annotator Guidelines

#### General Rules
- Read the entire ticket before labeling. Do not label based on the first sentence alone.
- Label based on the customer's primary intent, not secondary mentions.
- If a ticket contains multiple intents, label the one the customer is most urgently asking about.
- Do not infer intent from customer tone or sentiment. Focus on what they are asking for.

#### Edge Cases
| Scenario | Correct Label | Rationale |
|----------|-------------|-----------|
| "I was overcharged AND the feature is broken" | billing_issue | Financial issues take priority over product issues |
| "Cancel my account, your product is buggy" | cancellation | The ask is cancellation; the bug is context, not the request |
| "How do I export data before I cancel?" | cancellation | The underlying intent is preparing to leave |

#### Annotation Format
- **Input format**: Plain text (support ticket body, max 500 words)
- **Output format**: Single label from taxonomy
- **Tool**: Label Studio (self-hosted)

---

### 3. Quality Control

| QC Mechanism | Details |
|-------------|---------|
| Gold standard set size | 500 expert-labeled tickets |
| Gold standard check frequency | Every 100 items (5 gold items mixed in) |
| Minimum accuracy on gold set | 92% |
| Inter-rater reliability target | Cohen's kappa >= 0.85 |
| Review sampling rate | 15% of all annotations reviewed by lead |
| Dispute resolution process | Lead annotator makes final call; edge case added to guidelines |

#### Reliability Measurement
- **Metric**: Cohen's kappa (pairwise) and Fleiss' kappa (multi-annotator)
- **Measurement frequency**: Every 2,000 labeled items
- **Action if below threshold**: Pause labeling. Review disagreements. Update guidelines. Run a 200-item recalibration round.

---

### 4. Annotator Management

| Role | Count | Source | Training |
|------|-------|--------|----------|
| Lead annotator | 1 | In-house (senior support rep) | Defines gold standard with PM |
| Annotator | 4 | Vendor (Scale AI) | 2-hour guideline walkthrough + 100-item calibration |
| Reviewer | 1 | In-house (PM) | Reviews 15% sample, resolves disputes |

**Onboarding process**: 1) Read guidelines. 2) Label 100 calibration items. 3) Review disagreements with lead. 4) Must score 90%+ on calibration to proceed.
**Performance tracking**: Weekly accuracy, kappa, and throughput per annotator.
**Removal criteria**: Two consecutive weeks below 88% accuracy on gold items.

---

### 5. Semi-Automated Labeling

| Step | Details |
|------|---------|
| Pre-annotation model | GPT-4o-mini with few-shot classification prompt |
| Pre-annotation accuracy estimate | 78% (validated on 500-item pilot) |
| Human review rate | 100% of pre-annotations reviewed in Phase 1; drop to 30% after kappa > 0.90 |
| Correction tracking | Corrections logged; retrain prompt monthly with hardest examples |

---

### 6. Timeline and Budget

| Phase | Duration | Volume | Cost |
|-------|----------|--------|------|
| Guideline development | 3 days | N/A | 24 internal hours |
| Pilot labeling (calibration) | 5 days | 1,000 items | $800 |
| Production labeling | 6 weeks | 24,000 items | $14,400 |
| Quality audit | 3 days | 3,600 items reviewed | 24 internal hours |
| **Total** | **~8 weeks** | **25,000 items** | **$15,200 + 48 hrs internal** |

---

### 7. Feedback Loop

- [x] Labeling metrics tracked in weekly model performance dashboard
- [x] Model error analysis includes label audit on misclassified items
- [ ] Edge cases from model eval added to guidelines (quarterly update)
- [ ] Gold standard set refreshed with 50 new items per quarter

Key Takeaways

Invest heavily in annotator guidelines before labeling begins. Clear guidelines with edge cases prevent more errors than any QC process can catch after the fact.
Inter-rater reliability (kappa scores) is the single best predictor of labeling quality. Measure it early and often.
Semi-automated labeling with LLM pre-annotation can cut costs 30-50%, but requires 100% human review until you validate quality.
Run a calibration phase with a small pilot before production labeling. It reveals guideline gaps while they are still cheap to fix.
Connect labeling quality metrics to model performance metrics. If model accuracy drops, check labeling consistency before investigating model architecture.
Gold standard sets must evolve. Add new edge cases from production model failures to keep the gold set representative.

AI Data Labeling Template for AI Products