Optimizing DNN Compilation for Distributed Training with Joint OP and Tensor Fusion

Xiaodong Yi, Shiwei Zhang, Lansong Diao +6

This paper proposes DisCo, an automatic deep learning compilation module for data-parallel distributed training. Unlike most deep learning compilers that focus on training or infer…