Data-parallel training spends a surprising fraction of wall time not in matrix multiplies but in all-reduce: every GPU must average gradients before the optimizer step. Naive peer-to-peer copies or a parameter server become the bottleneck once you leave a single board. NVIDIA's NCCL (NVIDIA Collective Communications Library) provides topology-aware collectives — all-reduce, all-gather, reduce-scatter, broadcast — over NVLink, PCIe, NVSwitch, Ethernet, and InfiniBand. PyTorch Distributed, Horovod, and DeepSpeed all sit on it. The library's job is to pick rings or trees that match the machine, pipeline chunks, and keep SM time overlapping with the network.

Topology is the algorithm

On a DGX-class node, NVLink bandwidth dwarfs PCIe. A ring that hops the wrong way turns a 300 GB/s fabric into a PCIe crawl. NCCL probes the topology (and can be hinted with NCCL_IB_HCA, NCCL_P2P_LEVEL, and friends) to build rings that stay on the fast links. Multi-node jobs add a second level: intra-node reduce, then inter-node, then broadcast back — or hierarchical trees. Getting this wrong is the classic "eight GPUs slower than one" incident after a cluster networking change.

Chunking and CUDA streams let NCCL split a large tensor so communication overlaps the next kernel. Frameworks that issue a blocking all-reduce after the entire backward pass cannot hide latency. Gradient bucketing (PyTorch's default DDP) exists to make NCCL calls large enough to saturate links but early enough to overlap. Too-small tensors issue too many collectives; too-large waits for the whole backward.

Failures, hangs, and the watchdog

Collectives are tightly coupled. One rank slow, dead, or on a different NCCL version hangs the rest. Timeouts (NCCL_TIMEOUT) and framework watchdogs are how you get a stack trace instead of a silent 12-hour job. Mixed GPU generations in one communicator, or a container that sees a subset of devices differently on each rank (CUDA_VISIBLE_DEVICES mismatches), produce impossible topologies. NCCL logging (NCCL_DEBUG=INFO) is noisy and indispensable on first bring-up of a fabric.

Ethernet-only clusters can still run NCCL, with lower ceilings. GPU Direct RDMA and a properly tuned InfiniBand fabric are what large training jobs assume. Sharpened competition from RCCL on AMD and various one-sided networking stacks does not change the design lesson: the collective library is part of the machine, not an application detail. If you write your own ring in Python, you will lose to NCCL's bandwidth.

For serving, all-gather of sharded weights (inference-time tensor parallel) uses the same primitives. A chat with a 70B model split across GPUs is an NCCL problem as much as a kernel problem.

Also pin NCCL_SOCKET_IFNAME and the InfiniBand device list in Kubernetes: the wrong NIC — a management Ethernet while RDMA sits unused — is a common silent slowdown. Bring-up tests should print NCCL's chosen path and a known all-reduce bus bandwidth, then fail the job if the number is off by an order of magnitude from the SKU's spec.

What you can borrow

  • Use a topology-aware collective library; do not hand-roll rings in framework code.
  • Size and bucket reductions so they overlap compute and saturate the fast links you actually have.
  • Keep NCCL/framework versions identical across ranks; mixed stacks hang.
  • Set timeouts and dump NCCL_DEBUG on first hang — the failure is usually topology, visibility, or a dead rank.
  • Design jobs around the real hierarchy (NVLink vs. NIC). Benchmarks on a laptop PCIe GPU will not predict a switch fabric.