Data Pipeline Specification Template

The data engineer owns the technical implementation, but the product manager or data analyst who consumes the output should own the business requirements sections (Pipeline Overview, Output Specification, and quality thresholds). Shared ownership prevents pipelines that are technically correct but do not serve the business need.

Enough that a new team member could verify correctness without reading the code. Document business rules in plain language with specific examples. "Remove duplicates" is insufficient. "Deduplicate by (user_id, event_name, timestamp), keeping the earliest occurrence" is actionable. The [observability](/glossary/observability) glossary entry covers monitoring principles that apply here.

Always create a spec for pipelines that feed dashboards, models, or external consumers. Skip the spec only for throwaway exploratory queries. The cost of writing a spec is 30-60 minutes. The cost of debugging a production pipeline with no documentation is days.

Document expected schema in the Source Configuration section and add schema validation as a quality check. When source schemas change, the pipeline should fail loudly rather than silently producing wrong data. Include a runbook entry for schema migration in the Failure Handling section.

Start with your downstream consumer's needs and work backward. If the dashboard needs data by 8am, your pipeline needs to complete by 7:30am, which means it needs to start by 7:00am. Set freshness SLAs, completeness SLAs, and incident response SLAs separately. Use the [AI Readiness Assessment](/tools/ai-readiness-assessment) to evaluate whether your data infrastructure supports ML workloads. ---