GC3: An Optimizing Compiler for GPU Collective Communication

Meghan Cowan, Saeed Maleki, Madanlal Musuvathi +2

Machine learning models made up of millions or billions of parameters are trained and served on large multi-GPU systems. As models grow in size and execute on more GPUs, the collec…