Megatron-LM: Slicing Transformers Across GPUs on Purpose
NVIDIA's Megatron-LM paper showed tensor and pipeline parallelism so Transformer layers larger than one GPU's memory could still train efficiently on DGX-class boxes.
3 min read
Engineering at scale
4 articles · 12 min of reading
NVIDIA's Megatron-LM paper showed tensor and pipeline parallelism so Transformer layers larger than one GPU's memory could still train efficiently on DGX-class boxes.
TensorRT lowers trained networks into fused, precision-calibrated kernels so production inference can use Tensor Cores instead of running training graphs unchanged.
NCCL implements all-reduce and friends across GPUs and nodes so data-parallel training can use NVLink, NVSwitch, and InfiniBand without each framework rewriting rings.
NVIDIA's CUDA turned GPUs from graphics pipelines into a SIMT programming model that scientific computing and later neural networks could actually ship on.