NCCL: Collective Communication When One GPU Is Not Enough
NCCL implements all-reduce and friends across GPUs and nodes so data-parallel training can use NVLink, NVSwitch, and InfiniBand without each framework rewriting rings.
3 min read
Tag
2 articles · 6 min of reading
NCCL implements all-reduce and friends across GPUs and nodes so data-parallel training can use NVLink, NVSwitch, and InfiniBand without each framework rewriting rings.
NVIDIA's CUDA turned GPUs from graphics pipelines into a SIMT programming model that scientific computing and later neural networks could actually ship on.