Multi-Query Attention: One Key/Value Head, Many Queries, Smaller Decode Cache
Shazeer 2019 shared keys and values across attention heads to speed autoregressive inference. The quality trade versus full multi-head, and why GQA later filled the gap.
Tag
9 articles · 29 min of reading
Shazeer 2019 shared keys and values across attention heads to speed autoregressive inference. The quality trade versus full multi-head, and why GQA later filled the gap.
Kwon et al. (vLLM) allocate KV cache in pages, remap on the fly, and stop wasting giant contiguous buffers on every sequence. Throughput, sharing for prefixes, and the operational model of an LLM OS.
Leviathan et al. use a small draft model to propose token blocks and the large model to accept or reject them in parallel. Same distribution, fewer serial large-model steps, and the acceptance-rate math that decides whether you win.
Ainslie et al. interpolate between multi-head and multi-query attention by sharing key/value heads across groups of queries. The serving win, the quality trade, and how to uptrain an MHA checkpoint.
const type parameters (TypeScript 5.0): when T extends readonly any[] infers tuples of literals, and how that replaces as const on every call.
A framework for estimating LLM feature costs before shipping and the concrete levers that control spend once traffic scales beyond a prototype.
A practical breakdown of LLM quantization techniques, the real quality trade-offs behind them, and how to decide how far to push it.
The core techniques that make LLM inference fast and affordable at scale: batching, KV cache management, and speculative decoding explained.
A practical look at subword tokenization internals and the concrete ways tokenizer behavior affects cost, latency, and model quality.