tiktoken: Fast BPE That Keeps Product Token Counts Honest
OpenAI released tiktoken so applications could use the same byte-pair encoding as the API, making context limits and billing visible before a request is sent.
Tag
9 articles · 32 min of reading
OpenAI released tiktoken so applications could use the same byte-pair encoding as the API, making context limits and billing visible before a request is sent.
Databricks open-sourced MLflow so teams could record parameters, metrics, and artifacts from training runs without each group inventing a spreadsheet protocol.
Microsoft's Language Server Protocol split editor UI from language intelligence so a single server could power VS Code, Vim, and the rest of the industry.
Why Microsoft built TypeScript to add types to JavaScript incrementally, and how that bet scaled across Office web, VS Code, and the broader ecosystem.
How Google's internal Protocol Buffers serialization format and Stubby RPC framework evolved into gRPC, the open-source RPC standard many companies now run on.
A practical guide to using the React DevTools Profiler to find unnecessary re-renders and expensive commits before reaching for memoization.
How Google's internal Blaze build system, and its open-source descendant Bazel, made fast, reproducible builds possible across a codebase of billions of lines.
Why Stripe built its own type checker, Sorbet, instead of rewriting its large Ruby codebase, and how gradual typing let adoption happen incrementally.
Inside Piper and Client in the Cloud, the infrastructure that lets Google run a single monorepo at billions of lines of code without collapsing under its own weight.