Embedding Strategy Template for AI Products

There is no universal answer. Start with 256-512 tokens for most text content and adjust based on retrieval quality testing. Smaller chunks (128-256) work better when queries target specific facts. Larger chunks (512-1024) work better when queries need broader context. The right size depends on your content structure and how users phrase their queries.

Managed services (Pinecone, Weaviate Cloud, Qdrant Cloud) are the right default for most teams. They handle scaling, replication, and maintenance. Self-host only if you have strict data residency requirements, need to avoid vendor lock-in, or your team has strong infrastructure engineering capability. The operational cost of running a vector database is higher than most teams expect.

Build a test set of 50-100 representative queries with labeled relevant documents (the "ground truth" set). Measure precision@K (what fraction of returned results are relevant) and recall@K (what fraction of all relevant documents are returned). Run this evaluation before launch and on a weekly cadence after launch. The [AI Eval Scorecard](/tools/ai-eval-scorecard) provides a framework for structuring retrieval evaluations.

Add re-ranking when your precision@10 is good but your precision@3 (the results users actually read) needs improvement. Cross-encoder re-ranking models score each query-document pair individually, which is more accurate but slower. The typical pattern is: retrieve top 20-50 via vector search (fast), then re-rank to find the best 3-5 (slower but more accurate). Re-ranking adds 50-200ms of latency and model API cost per query.

Build an incremental indexing pipeline that listens for content changes (webhooks, change data capture, or polling). When a document is updated, re-chunk and re-embed only that document. Use the document_id metadata field to find and replace stale vectors. Schedule a full re-index monthly or quarterly as a consistency check. The [RAG Architecture Template](/templates/rag-architecture-template) covers pipeline design in more detail.