Multi-Query Attention: One Key/Value Head, Many Queries, Smaller Decode Cache
Shazeer 2019 shared keys and values across attention heads to speed autoregressive inference. The quality trade versus full multi-head, and why GQA later filled the gap.
Tag
12 articles · 40 min of reading
Shazeer 2019 shared keys and values across attention heads to speed autoregressive inference. The quality trade versus full multi-head, and why GQA later filled the gap.
Su et al. rotate query and key pairs by an angle that depends on position, encoding relative offset in the dot product. Why LLaMA-style models adopted it, and what breaks at lengths you did not train.
Ainslie et al. interpolate between multi-head and multi-query attention by sharing key/value heads across groups of queries. The serving win, the quality trade, and how to uptrain an MHA checkpoint.
Radford et al. trained a single encoder-decoder on 680k hours of noisy web audio and many languages. Why weakly labeled scale beat many specialized ASR stacks, and where it still fails.
Raffel et al. cast translation, classification, and summarization as string-to-string tasks on a masked span-filling pretrained encoder-decoder. The framework still hiding behind a lot of seq2seq fine-tunes.
Dosovitskiy et al. dropped the conv inductive bias, split images into patches, and trained a transformer on ImageNet-21k-scale data. When ViTs beat ResNets, and when they need the extra data or distillation.
How top-k expert routing works, why load balancing losses exist, and the serving problems (expert parallelism, tail latency) the Switch Transformer paper made unavoidable.
Dao et al. showed that attention was memory-bound, not math-bound. Here is the IO story and what it changed for long-context training.
Masked language modeling versus left-to-right GPT, next-sentence prediction, and what still matters when you fine-tune encoder models in products.
A field reading of Vaswani et al.: why dropping recurrence unlocked GPU training, and which pieces of the original transformer teams still keep.
A conceptual walkthrough of what attention mechanisms actually do, working up from the original translation problem they were built to solve.
A practical walkthrough of self-attention, multi-head attention, and positional encoding for engineers who need to reason about transformers, not just use them.