Optimizing Allreduce Operations for Modern Heterogeneous Architectures with Multiple Processes per GPU

Michael Adams, Amanda Bienz

Large inter-GPU all-reduce operations, prevalent throughout deep learning, are bottlenecked by communication costs. Emerging heterogeneous architectures are comprised of complex no…