On the Ineffectiveness of Variance Reduced Optimization for Deep Learning
arXiv:1812.04529
Abstract
The application of stochastic variance reduction to optimization has shown remarkable recent theoretical and practical success. The applicability of these techniques to the hard non-convex optimization problems encountered during training of modern deep neural networks is an open problem. We show that naive application of the SVRG technique and related approaches fail, and explore why.
References in corpus (9)
- Adam: A Method for Stochastic Optimization
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Wide Residual Networks
- A Universal Catalyst for First-Order Optimization
- Non-convex Finite-Sum Optimization Via SCSG Methods
- Finito: A Faster, Permutable Incremental Gradient Method for Big Data Problems
- A Simple Practical Accelerated Method for Finite Sums
- Stochastic Optimization with Variance Reduction for Infinite Datasets with Finite-Sum Structure
Cited by in corpus (7)
- Don't Use Large Mini-Batches, Use Local SGD
- Reducing Noise in GAN Training with Variance Reduced Extragradient
- Distributed Learning of Deep Neural Networks using Independent Subnet Training
- Anarchic Federated Learning
- On the Convergence of SARAH and Beyond
- Bi-fidelity Stochastic Gradient Descent for Structural Optimization under Uncertainty
- Towards Better Generalization: BP-SVRG in Training Deep Neural Networks