Knowledge Distillation: Train the Student on the Teacher's Soft Distribution
Hinton, Vinyals, and Dean taught small models to match a large model's temperature-smoothed probabilities. Why logits transfer more than hard labels, and how distillation still ships in 2026 stacks.
3 min read