tiktoken: Fast BPE That Keeps Product Token Counts Honest
OpenAI released tiktoken so applications could use the same byte-pair encoding as the API, making context limits and billing visible before a request is sent.
Tag
6 articles · 19 min of reading
OpenAI released tiktoken so applications could use the same byte-pair encoding as the API, making context limits and billing visible before a request is sent.
How OpenAI turned a sequence of larger GPT models into a versioned HTTP API with tokens, rate limits, and compatibility constraints that application teams could build on.
How Square's Payments API uses idempotency keys so retries from POS apps and readers do not double-charge cards.
JEP 431: SequencedCollection, SequencedMap, reversed views, and how new addFirst/getLast methods change APIs without a new List subtype zoo.
How Stripe's idempotency key design lets clients safely retry payment requests after a timeout without risking a duplicate charge.
How LinkedIn's move to a service-oriented architecture forced a standardized REST framework, and why schema-driven API contracts paid off at scale.