On the Ineffectiveness of Variance Reduced Optimization for Deep Learning

Abstract

The application of stochastic variance reduction to optimization has shown remarkable recent theoretical and practical success. The applicability of these techniques to the hard non-convex optimization problems encountered during training of modern deep neural networks is an open problem. We show that naive application of the SVRG technique and related approaches fail, and explore why.

References in corpus (2)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Finito: A Faster, Permutable Incremental Gradient Method for Big Data Problems

Cited by in corpus (7)

• Don't Use Large Mini-Batches, Use Local SGD
• Reducing Noise in GAN Training with Variance Reduced Extragradient
• Distributed Learning of Deep Neural Networks using Independent Subnet Training
• Anarchic Federated Learning
• On the Convergence of SARAH and Beyond
• Bi-fidelity Stochastic Gradient Descent for Structural Optimization under Uncertainty
• Towards Better Generalization: BP-SVRG in Training Deep Neural Networks