Definition
A token is the fundamental unit of text that large language models process. Before an LLM can read or generate text, the input must be broken into tokens by a tokenizer. Each model family uses its own tokenizer: GPT-4 uses cl100k_base, Claude uses a custom BPE tokenizer, and Llama uses SentencePiece. These tokenizers split text into subword units that balance vocabulary size with the ability to represent any text.
Common English words are usually single tokens ("the", "product", "feature"). Longer or less common words get split into multiple tokens ("tokenization" might become "token" + "ization"). Numbers, code, and non-English text often require more tokens per character. The OpenAI tokenizer tool (tiktoken) and Anthropic's token counter are useful for estimating costs before building features.
Tokens define two critical constraints for AI products: the context window (maximum tokens a model can process in one request) and the cost (API pricing is per-token). Understanding tokens helps PMs set usage limits, estimate infrastructure costs, and design prompts that maximize value within constraints. You can model how token costs affect product economics using the LLM Cost Estimator.
Why It Matters for Product Managers
Tokens are the unit of cost for AI products, much like compute hours for cloud infrastructure or API calls for integration-heavy products. Every PM building with LLMs needs to understand tokens because they directly control contribution margin. A feature that sends 10,000 tokens per request costs 10x more than one that achieves the same result in 1,000 tokens.
Token economics also shape product design. If your context window is 128K tokens, you can include a user's entire document history in each request. If it is 8K tokens, you need a retrieval strategy (like RAG) to select only the most relevant context. These are product architecture decisions that affect user experience, cost, and latency.
How to Apply It
Build token awareness into your product development process from day one. Treating tokens as an afterthought leads to cost surprises that force painful feature rollbacks or pricing changes.
Practical steps for PMs:
- ☐ Estimate token usage per feature before building (input prompt + expected output + conversation context)
- ☐ Set up token usage monitoring and dashboards segmented by feature and user tier
- ☐ Implement prompt engineering best practices to reduce token waste in system prompts
- ☐ Choose model tiers based on task complexity (cheap models for routing, expensive models for generation)
- ☐ Set max_tokens limits on outputs to prevent runaway costs from verbose responses
- ☐ Build token budgets into your pricing model so heavy users pay proportionally