ZeRO: Shard Optimizer State, Gradients, and Parameters Instead of Replicating Them
Rajbhandari et al. (ZeRO) split Adam's memory across data-parallel ranks so you can train models that would OOM if every GPU held a full copy. Stages 1–3, communication volume, and when not to go to ZeRO-3.
3 min read