Mixtral: Sparse Experts in a Dense-Looking Decoder You Can Actually Serve
Jiang et al. replace some MLP blocks with a Mixture-of-Experts router (Mixtral 8×7B). Active parameters per token stay modest; capacity and serving constraints do not. How to think about MoE without the hype.
3 min read