Toward Model Parallelism for Deep Neural Network based on Gradient-free ADMM Framework

Abstract

Alternating Direction Method of Multipliers (ADMM) has recently been proposed as a potential alternative optimizer to the Stochastic Gradient Descent(SGD) for deep learning problems. This is because ADMM can solve gradient vanishing and poor conditioning problems. Moreover, it has shown good scalability in many large-scale deep learning applications. However, there still lacks a parallel ADMM computational framework for deep neural networks because of layer dependency among variables. In this paper, we propose a novel parallel deep learning ADMM framework (pdADMM) to achieve layer parallelism: parameters in each layer of neural networks can be updated independently in parallel. The convergence of the proposed pdADMM to a critical point is theoretically proven under mild conditions. The convergence rate of the pdADMM is proven to be $o(1/k)$ where $k$ is the number of iterations. Extensive experiments on six benchmark datasets demonstrated that our proposed pdADMM can lead to more than 10 times speedup for training large-scale deep neural networks, and outperformed most of the comparison methods. Our code is available at: https://github.com/xianggebenben/pdADMM.

ICDM2020

References in corpus (12)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• ADADELTA: An Adaptive Learning Rate Method
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Horovod: fast and easy distributed deep learning in TensorFlow
• Training Neural Networks Without Gradients: A Scalable ADMM Approach
• Training Neural Networks Using Features Replay
• Adaptive Consensus ADMM for Distributed Optimization
• Differentiable Linearized ADMM
• Fully Decoupled Neural Network Learning Using Delayed Gradients
• Predict Globally, Correct Locally: Parallel-in-Time Optimal Control of Neural Networks
• The Application of Multi-block ADMM on Isotonic Regression Problems
• Tunable Subnetwork Splitting for Model-parallelism of Neural Network Training