tiktoken: Fast BPE That Keeps Product Token Counts Honest
OpenAI released tiktoken so applications could use the same byte-pair encoding as the API, making context limits and billing visible before a request is sent.
3 min read
Engineering at scale
3 articles · 9 min of reading
OpenAI released tiktoken so applications could use the same byte-pair encoding as the API, making context limits and billing visible before a request is sent.
OpenAI's Whisper paper turned weakly supervised web audio into a multilingual transcriber; production use then had to solve chunking, latency, and punctuation as a serving problem.
How OpenAI turned a sequence of larger GPT models into a versioned HTTP API with tokens, rate limits, and compatibility constraints that application teams could build on.