EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks

Shengwei Li, Zhiquan Lai, Dongsheng Li +3

Distributed data-parallel training has been widely adopted for deep neural network (DNN) models. Although current deep learning (DL) frameworks scale well for dense models like ima…