HOCCL: Offloading Collective Communication from GPU Cores to Accelerate Distributed Training

Yao Fei, Gongming Zhao, Hongli Xu +5

Large language model training involves massive computation on GPU streaming multiprocessors (SMs), the primary compute units of GPUs. Since SMs host specialized accelerators such a…