At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?

Abstract

Background: Recent developments have made it possible to accelerate neural networks training significantly using large batch sizes and data parallelism. Training in an asynchronous fashion, where delay occurs, can make training even more scalable. However, asynchronous training has its pitfalls, mainly a degradation in generalization, even after convergence of the algorithm. This gap remains not well understood, as theoretical analysis so far mainly focused on the convergence rate of asynchronous methods. Contributions: We examine asynchronous training from the perspective of dynamical stability. We find that the degree of delay interacts with the learning rate, to change the set of minima accessible by an asynchronous stochastic gradient descent algorithm. We derive closed-form rules on how the learning rate could be changed, while keeping the accessible set the same. Specifically, for high delay values, we find that the learning rate should be kept inversely proportional to the delay. We then extend this analysis to include momentum. We find momentum should be either turned off, or modified to improve training stability. We provide empirical experiments to validate our theoretical findings.

arXiv commentICLR 2020 Camera ready version

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Wide Residual Networks
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Revisiting Distributed Synchronous SGD
• Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
• Measuring the Effects of Data Parallelism on Neural Network Training
• Stochastic Gradient Push for Distributed Deep Learning
• A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Order
• Asynchronous Decentralized Parallel Stochastic Gradient Descent
• A Tight Convergence Analysis for Stochastic Gradient Descent with Delayed Updates
• Toward Understanding the Impact of Staleness in Distributed Machine Learning
• Taming Momentum in a Distributed Asynchronous Environment

Cited by in corpus (4)

• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
• Pipelined Backpropagation at Scale: Training Large Models without Batches
• DBS: Dynamic Batch Size For Distributed Deep Neural Network Training
• Acceleration via Fractal Learning Rate Schedules