Distributed stochastic optimization for deep learning (thesis)

Abstract

We study the problem of how to distribute the training of large-scale deep learning models in the parallel computing environment. We propose a new distributed stochastic optimization method called Elastic Averaging SGD (EASGD). We analyze the convergence rate of the EASGD method in the synchronous scenario and compare its stability condition with the existing ADMM method in the round-robin scheme. An asynchronous and momentum variant of the EASGD method is applied to train deep convolutional neural networks for image classification on the CIFAR and ImageNet datasets. Our approach accelerates the training and furthermore achieves better test accuracy. It also requires a much smaller amount of communication than other common baseline approaches such as the DOWNPOUR method. We then investigate the limit in speedup of the initial and the asymptotic phase of the mini-batch SGD, the momentum SGD, and the EASGD methods. We find that the spread of the input data distribution has a big impact on their initial convergence rate and stability region. We also find a surprising connection between the momentum SGD and the EASGD method with a negative moving average rate. A non-convex case is also studied to understand when EASGD can get trapped by a saddle point. Finally, we scale up the EASGD method by using a tree structured network topology. We show empirically its advantage and challenge. We also establish a connection between the EASGD and the DOWNPOUR method with the classical Jacobi and the Gauss-Seidel method, thus unifying a class of distributed stochastic optimization methods.

arXiv commentThis is the author's thesis at under supervision of Yann LeCun. Part of the results are based on the paper arXiv:1412.6651 in collaboration with Anna Choromanska and Yann LeCun

References in corpus (11)

• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• The Loss Surfaces of Multilayer Networks
• Optimal Distributed Online Prediction using Mini-Batches
• Convolutional neural networks with low-rank regularization
• Train faster, generalize better: Stability of stochastic gradient descent
• Slow Learners are Fast
• Staleness-aware Async-SGD for Distributed Deep Learning
• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• SparkNet: Training Deep Networks in Spark
• Fundamental Limits of Online and Distributed Algorithms for Statistical Learning and Estimation
• GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network Training