Large language APIs bill and bound work in tokens, not characters. A naive len(s) in Python will not tell you whether a prompt fits in 8k or 128k, and it will not match the invoice. Byte-pair encoding (BPE), long used in GPT-style models, merges frequent byte pairs into a vocabulary that handles any Unicode string without an UNK while compressing common English subwords. OpenAI's tiktoken is a fast BPE implementation with the actual vocabularies (cl100k_base and others) used by production models. Shipping it as a library turned tokenization from a research detail into an application-layer primitive: truncate, chunk for retrieval, estimate cost, and reject oversize requests locally.
Encoding is a compatibility surface
If your local tokenizer disagrees with the server, you will either fail at the API with a context error or silently drop text you thought you sent. tiktoken exists so the client and the service share merge rules and special tokens. Model families do not share encodings forever; counting with the wrong encoding is a bug that looks like "the model ignored the end of my prompt." Product code should map model_id → encoding_name in one module, not scatter tiktoken.get_encoding("cl100k_base") with a comment that was true in 2023.
BPE's behavior on whitespace, punctuation, and languages with large character sets surprises people. A single CJK character may be one token or more depending on the vocab; a JSON blob with lots of braces can be surprisingly expensive. That is why cost dashboards should show tokens, not kilobytes. Chunking RAG documents by character length produces uneven token chunks and uneven retrieval quality. Chunk by tokens.
Performance and special tokens
Python loops over BPE merges would be too slow for high-QPS gateways. tiktoken uses optimized cores so counting a long prompt is cheap compared to the network call. Gateways should still cache counts for identical system prompts. Special tokens (<|endoftext|> and chat templates) are where DIY tokenizers go wrong: wrapping a chat transcript with the wrong role tokens changes both count and model behavior. Prefer the platform's official chat formatting helpers when they exist; tiktoken counts the string you will actually send.
Security belongs here too. Tokenizers can be stressed with pathological Unicode; more importantly, user text that includes special-token-like sequences should be treated as data. Injection via prompt is a product issue, but leaking or honoring raw special tokens from users is a tokenizer-integration issue.
The deeper steal is to make the unit of the backend visible at the edge of your app. If the scarce resource is tokens, every queue, UI counter, and truncation policy should speak tokens. tiktoken is the ruler.
Truncation policy deserves an explicit product choice: drop oldest conversation turns, drop retrieved chunks first, or refuse the request. Silent truncation of system instructions is how assistants "forget" safety and style. Count the reserved prefix (system + tools) separately from the user budget so a long tool schema cannot eat the user's question. That bookkeeping is dull and it is the difference between a predictable API wrapper and a heisenbug in production.
What you can borrow
- Count and chunk with the same tokenizer the model uses; never use character length for context budgets.
- Centralize the map from model ID to encoding; encodings change across families.
- Measure RAG chunk size in tokens so retrieval batches stay even.
- Include chat templates and special tokens in the count you compare to the context window.
- Show token estimates in developer tooling and billing UI so cost is not a month-end surprise.