Adam: Adaptive Moments That Became the Default Optimizer — With the Caveats Intact
Kingma and Ba combined RMSProp-style second moments with momentum and a bias correction. Why Adam trains transformers, and when the paper's own advice still says to try SGD.
3 min read