Stochastic Nonconvex Optimization with Large Minibatches
arXiv:1709.08728
Abstract
We study stochastic optimization of nonconvex loss functions, which are typical objectives for training neural networks. We propose stochastic approximation algorithms which optimize a series of regularized, nonlinearized losses on large minibatches of samples, using only first-order gradient information. Our algorithms provably converge to an approximate critical point of the expected objective with faster rates than minibatch stochastic gradient descent, and facilitate better parallelization by allowing larger minibatches.
arXiv commentAccepted by the ALT 2019
References in corpus (14)
- ADADELTA: An Adaptive Learning Rate Method
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Optimal Distributed Online Prediction using Mini-Batches
- Train longer, generalize better: closing the generalization gap in large batch training of neural networks
- Communication Efficient Distributed Optimization using an Approximate Newton-type Method
- Convergence Rates of Inexact Proximal-Gradient Methods for Convex Optimization
- A Universal Catalyst for First-Order Optimization
- AIDE: Fast and Communication Efficient Distributed Optimization
- Accelerated Methods for Non-Convex Optimization
- Distributed Stochastic Variance Reduced Gradient Methods and A Lower Bound for Communication Complexity
- Natasha: Faster Non-Convex Stochastic Optimization Via Strongly Non-Convex Parameter
- Incremental Aggregated Proximal and Augmented Lagrangian Algorithms
- Memory and Communication Efficient Distributed Stochastic Optimization with Minibatch-Prox
- Without-Replacement Sampling for Stochastic Gradient Methods: Convergence Results and Application to Distributed Optimization
Cited by in corpus (8)
- Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
- A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning
- Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training
- Distributed Machine Learning for Wireless Communication Networks: Techniques, Architectures, and Applications
- Extrapolation for Large-batch Training in Deep Learning
- Gradient Sparification for Asynchronous Distributed Training
- Stochastic Gradient Descent for Stochastic Doubly-Nonconvex Composite Optimization
- Stochastic Proximal Gradient Algorithm with Minibatches. Application to Large Scale Learning Models