RedSync : Reducing Synchronization Traffic for Distributed Deep Learning

Jiarui Fang, Haohuan Fu, Guangwen Yang +1

Data parallelism has become a dominant method to scale Deep Neural Network (DNN) training across multiple nodes. Since synchronizing a large number of gradients of the local model…