What is a good target for CI pipeline duration?

Under 10 minutes for PR checks, under 15 minutes for merge builds. Google's research on developer productivity shows that build times over 10 minutes significantly reduce the number of daily integrations per developer. Under 5 minutes is excellent and enables true continuous integration. Track your P50 and P95 separately since P95 catches the occasional slow run that frustrates developers.

Should we invest in faster runners or smarter pipelines?

Start with pipeline optimization (caching, parallelization, selective execution). These give you 50-80% improvement at zero incremental cost. Faster runners give you linear improvement (2x faster runner = 2x faster build) but at higher cost. Use both: optimize the pipeline first, then right-size runners for the remaining workload. Track [pipeline efficiency](/glossary/prioritization) as a team health metric.

How do we prevent pipeline performance from regressing?

Set alerting thresholds on P50 and P95 duration. When a PR increases pipeline duration by more than 10%, flag it in the PR review. Run a weekly "top 10 slowest tests" report and assign owners to investigate. Treat pipeline performance like production latency: it needs ongoing monitoring, not a one-time fix.

How do we handle flaky tests in CI?

Quarantine flaky tests into a separate non-blocking test suite. Track the quarantine list and assign owners. Set a policy: flaky tests get 1 week to be fixed or deleted. Flaky tests that block CI erode trust in the pipeline. Engineers start ignoring failures, and real bugs slip through. The [Technical PM Handbook](/technical-pm-guide) covers engineering health metrics including flaky test rates.

Is it worth parallelizing a test suite under 5 minutes?

Usually not. The overhead of splitting, distributing, and merging results adds 30-60 seconds. If your suite takes 4 minutes, parallelizing into 4 shards might reduce it to 2 minutes. The effort to maintain time-balanced sharding probably is not worth the 2-minute gain unless you run hundreds of builds per day. ---

CI/CD Pipeline Optimization Template

What This Template Is For

A slow CI pipeline is a tax on every engineering decision. When builds take 40 minutes, developers batch changes, avoid running tests locally, and context-switch while waiting. The result is larger pull requests, slower feedback, and more production incidents.

Most CI pipelines are not slow because of a single bottleneck. They accumulate slowness over time: a test suite that grew from 2 minutes to 15, a Docker build that downloads the same dependencies every run, a linting step that checks the entire codebase when only three files changed.

This template provides a systematic approach to auditing and optimizing CI/CD performance. It is not about switching CI providers or adopting a new tool. It is about measuring where time is spent, identifying the highest-impact optimizations, and tracking improvement over time. Use it when your team complains about build times, when your CI costs are growing faster than your engineering team, or when you want to improve developer experience. For the broader CI/CD specification, use the CI/CD Pipeline Template. For context on developer productivity metrics, see the Technical PM Handbook.

How to Use This Template

Collect baseline metrics: measure current build times for your three most common pipeline types (PR check, merge to main, production deploy).
Break down each pipeline into stages and measure the duration of each. Identify the top three time-consuming stages.
For each slow stage, document the root cause (dependency install, test execution, Docker build, etc.) and the candidate optimization.
Prioritize optimizations by impact (minutes saved per run x runs per day) and effort (hours of engineering work).
Implement optimizations one at a time, measuring the before/after impact of each.
Set up ongoing monitoring so pipeline performance does not regress.

The Template

Pipeline Performance Baseline

Pipeline Type	Trigger	Current P50 Duration	Current P95 Duration	Runs/Day	Target P50
PR Check	PR open/update	[min]	[min]	[count]	[min]
Merge Build	Merge to main	[min]	[min]	[count]	[min]
Production Deploy	Tag/manual	[min]	[min]	[count]	[min]

Cost baseline.

Metric	Current	Target
Monthly CI compute cost	[$]	[$]
Avg cost per pipeline run	[$]	[$]
Compute minutes/month	[min]	[min]

Stage-by-Stage Breakdown

Pipeline: [PR Check]

Stage	Duration (P50)	Duration (P95)	% of Total	Parallelizable?
Checkout	[s]	[s]	[%]	No
Install dependencies	[s]	[s]	[%]	No
Lint	[s]	[s]	[%]	Yes
Type check	[s]	[s]	[%]	Yes
Unit tests	[s]	[s]	[%]	Yes
Integration tests	[s]	[s]	[%]	Yes
Build	[s]	[s]	[%]	No
Security scan	[s]	[s]	[%]	Yes
Total	[s]	[s]	100%

Optimization Opportunities

Category 1: Caching

Cache Target	Current State	Optimization	Expected Savings
Dependencies (node_modules, pip)	[No cache / Partial]	[Dependency lockfile cache]	[60-120s/run]
Build output (.next, dist)	[No cache]	[Incremental build cache]	[30-90s/run]
Docker layers	[No layer cache]	[BuildKit cache mount + registry cache]	[60-180s/run]
Test fixtures	[Generated each run]	[Cache synthetic test data]	[10-30s/run]

Cache configuration checklist.

☐ Dependency cache keyed on lockfile hash (package-lock.json, yarn.lock, poetry.lock)
☐ Build cache keyed on source file hash (not timestamp)
☐ Cache restored before install step (not after)
☐ Cache invalidated on major version bumps (Node, Python, etc.)
☐ Cache size monitored (large caches can slow restore more than they save)

Category 2: Test Parallelization

Test Suite	Current Duration	Shards/Workers	Expected Duration	Splitting Strategy
Unit tests	[min]	[N shards]	[min]	[File-based / Time-based]
Integration tests	[min]	[N shards]	[min]	[File-based]
E2E tests	[min]	[N shards]	[min]	[Time-based balancing]

Test optimization checklist.

☐ Tests split by historical execution time (not file count) for balanced shards
☐ Test order randomized (detect order-dependent tests)
☐ Slow test identified and tagged (tests > 10s individually reviewed)
☐ Flaky tests quarantined (tracked separately, not blocking)
☐ Test data generated in parallel (not sequentially before tests)

Category 3: Selective Execution

Optimization	Trigger Condition	Stages Skipped	Expected Savings
Skip E2E on docs-only changes	Only .md files changed	E2E, Build	[5-10 min]
Skip backend tests on frontend changes	Only src/ui/ changed	Backend unit, integration	[3-8 min]
Skip full build on test-only changes	Only __tests__/ changed	Build, Deploy	[2-5 min]

Path filter configuration.

# Example: skip backend tests when only frontend files changed
paths-filter:
  frontend: 'src/ui/**'
  backend: 'src/api/**'
  docs: '**/*.md'
  config: '.github/**'

# Run backend tests only when backend paths change
# Run all tests when config paths change

Category 4: Docker Build Optimization

Optimization	Description	Expected Savings
Multi-stage builds	Separate build and runtime stages	[Image size: -60%]
Layer ordering	Copy package files before source code	[Cache hit rate: +80%]
.dockerignore	Exclude node_modules, .git, tests from context	[Context transfer: -90%]
BuildKit cache	Mount dependency cache in build step	[Install: -70%]
Base image	Switch from node:18 to node:18-slim or distroless	[Image size: -50%]

Optimization Priority Matrix

Optimization	Impact (min saved/run)	Effort (hours)	Runs/Day	Daily Savings (min)	Priority
[Cache dependencies]	[2 min]	[2h]	[50]	[100 min]	[P1]
[Parallelize unit tests]	[3 min]	[4h]	[50]	[150 min]	[P1]
[Selective execution]	[5 min]	[8h]	[30]	[150 min]	[P1]
[Docker layer cache]	[2 min]	[3h]	[20]	[40 min]	[P2]
[Incremental build]	[1 min]	[6h]	[50]	[50 min]	[P2]

Ongoing Monitoring

Metric	Dashboard	Alert Threshold
PR check P50 duration	[URL]	> [target] min
PR check P95 duration	[URL]	> [target] min
Cache hit rate	[URL]	< [80%]
Flaky test rate	[URL]	> [2%] of runs
Monthly CI cost	[URL]	> [$target]
Queue wait time	[URL]	> [5 min] (insufficient runners)

☐ Pipeline duration tracked per stage over time (detect gradual regression)
☐ Alert on P95 duration exceeding target (catches tail latency issues)
☐ Weekly report on top 10 slowest tests
☐ Monthly CI cost review with cost-per-engineer metric
☐ Quarterly optimization review (re-audit top bottlenecks)

Filled Example: Monorepo Optimization (40 min to 12 min)

Baseline

Pipeline	Before P50	After P50	Improvement
PR Check	38 min	11 min	-71%
Merge Build	42 min	14 min	-67%
Deploy	25 min	18 min	-28%

Optimizations Applied (in priority order)

#	Optimization	Savings	Effort	Details
1	Dependency cache (npm ci)	-4 min	1h	Cache `~/.npm` keyed on `package-lock.json` hash
2	Parallelize tests (4 shards)	-12 min	3h	Jest `--shard` flag, time-based balancing via `jest-slow-test-detector`
3	Path-based filtering	-8 min	4h	Skip E2E on docs/config changes, skip backend on frontend-only PRs
4	Docker BuildKit cache	-3 min	2h	`--mount=type=cache,target=/root/.npm` in Dockerfile
5	Next.js incremental cache	-2 min	1h	Persist `.next/cache` between runs

Total: -29 minutes (76% reduction), 11 hours of engineering effort.

Cost impact. Monthly CI spend dropped from $2,800 to $1,100 (-61%) by reducing compute minutes and enabling spot instances for parallelized test shards.

Key Takeaways

Measure before optimizing: break pipelines into stages and identify the top three bottlenecks
Caching dependencies and build outputs is typically the highest-impact, lowest-effort win
Parallelize tests using time-based balancing, not just file count
Skip irrelevant stages based on changed file paths to cut unnecessary work
Monitor pipeline duration continuously and alert on regression

About This Template

Created by: Tim Adair

Last Updated: 3/5/2026

Version: 1.0.0

License: Free for personal and commercial use

CI/CD Pipeline Optimization Template