Stochastic Nonconvex Optimization with Large Minibatches
arXiv:1709.08728
Abstract
We study stochastic optimization of nonconvex loss functions, which are typical objectives for training neural networks. We propose stochastic approximation algorithms which optimize a series of regularized, nonlinearized losses on large minibatches of samples, using only first-order gradient information. Our algorithms provably converge to an approximate critical point of the expected objective with faster rates than minibatch stochastic gradient descent, and facilitate better parallelization by allowing larger minibatches.
Accepted by the ALT 2019
References in corpus (7)
- ADADELTA: An Adaptive Learning Rate Method
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Train longer, generalize better: closing the generalization gap in large batch training of neural networks
- AIDE: Fast and Communication Efficient Distributed Optimization
- Accelerated Methods for Non-Convex Optimization
- Natasha: Faster Non-Convex Stochastic Optimization Via Strongly Non-Convex Parameter
- Memory and Communication Efficient Distributed Stochastic Optimization with Minibatch-Prox
Cited by in corpus (8)
- Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
- A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning
- Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training
- Distributed Machine Learning for Wireless Communication Networks: Techniques, Architectures, and Applications
- Extrapolation for Large-batch Training in Deep Learning
- Gradient Sparification for Asynchronous Distributed Training
- Stochastic Gradient Descent for Stochastic Doubly-Nonconvex Composite Optimization
- Stochastic Proximal Gradient Algorithm with Minibatches. Application to Large Scale Learning Models