Priority-based Parameter Propagation for Distributed DNN Training

Abstract

Data parallel training is widely used for scaling distributed deep neural network (DNN) training. However, the performance benefits are often limited by the communication-heavy parameter synchronization step. In this paper, we take advantage of the domain specific knowledge of DNN training and overlap parameter synchronization with computation in order to improve the training performance. We make two key observations: (1) the optimal data representation granularity for the communication may differ from that used by the underlying DNN model implementation and (2) different parameters can afford different synchronization delays. Based on these observations, we propose a new synchronization mechanism called Priority-based Parameter Propagation (P3). P3 synchronizes parameters at a finer granularity and schedules data transmission in such a way that the training process incurs minimal communication delay. We show that P3 can improve the training throughput of ResNet-50, Sockeye and VGG-19 by as much as 25%, 38% and 66% respectively on clusters with realistic network bandwidth

In proceedings of the 2nd SysML Conference 2019

References in corpus (6)

• Deep Residual Learning for Image Recognition
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Revisiting Distributed Synchronous SGD
• Sockeye: A Toolkit for Neural Machine Translation
• GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent

Cited by in corpus (10)

• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Communication optimization strategies for distributed deep neural network training: A survey
• DAPPLE: A Pipelined Data Parallel Approach for Training Large Models
• Dynamic Parameter Allocation in Parameter Servers
• MicroRec: Efficient Recommendation Inference by Hardware and Data Structure Solutions
• Elastic Consistency: A General Consistency Model for Distributed Stochastic Gradient Descent
• EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks
• Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training
• Comprehensive Deadlock Prevention for GPU Collective Communication
• Scheduling Optimization Techniques for Neural Network Training