mKernel: Fast Multi-GPU, Multi-Node Fused Kernels

Ziming Mao, Yihan Zhang, Shawn Wei Chew +5

Communication has become a bottleneck in distributed training and inference of large models. Overlapping communication with computation at the granularity of kernels, on separate s…