Priority-based Parameter Propagation for Distributed DNN Training

Anand Jayarajan, Jinliang Wei, Garth Gibson +2

Data parallel training is widely used for scaling distributed deep neural network (DNN) training. However, the performance benefits are often limited by the communication-heavy par…