Kaplan Scaling Laws: Loss as a Power Law in Size, Data, and Compute
Kaplan et al. 2020 fitted smooth power laws for transformer language-model loss. What the laws still help you plan, and which extrapolation the Chinchilla generation had to unwind.
Tag
9 articles · 26 min of reading
Kaplan et al. 2020 fitted smooth power laws for transformer language-model loss. What the laws still help you plan, and which extrapolation the Chinchilla generation had to unwind.
Hoffmann et al. showed Kaplan-style laws overemphasized parameters. For a fixed FLOP budget, smaller models trained on more tokens win. How to use the 20-token rule without cargo-culting 2022 hardware.
Chowdhery et al. trained a 540B dense decoder with Pathways on TPU pods. Discontinuous jumps on reasoning, multilinguality, and the infrastructure story behind a single giant dense model.
Touvron et al. trained dense transformers from 7B to 65B on a documented public-data mix and showed smaller models could match larger proprietary ones when you spend tokens well. Why the data mix and tokenizer became the real API.
Schick et al. let a language model decide where API calls would help, executed the calls, and fine-tuned on the traces that actually lowered loss. A recipe for teaching tools without full RL.
Wei et al. showed that a few worked solutions in the prompt, with explicit reasoning traces, lift large models on math and symbolic tasks. When CoT is load-bearing and when it is theater.
Brown et al. showed that a 175B decoder, prompted with a handful of examples, could do tasks no one fine-tuned it for. What that changed for APIs, eval, and when prompting is the wrong tool.
Radford et al. 2019 trained a decoder on WebText and showed zero-shot translation, QA, and summarization from next-token prediction. The interface shift that GPT-3 later scaled.
Jiang et al. replace some MLP blocks with a Mixture-of-Experts router (Mixtral 8×7B). Active parameters per token stay modest; capacity and serving constraints do not. How to think about MoE without the hype.