Towards Better Generalization: BP-SVRG in Training Deep Neural Networks

Abstract

Stochastic variance-reduced gradient (SVRG) is a classical optimization method. Although it is theoretically proved to have better convergence performance than stochastic gradient descent (SGD), the generalization performance of SVRG remains open. In this paper we investigate the effects of some training techniques, mini-batching and learning rate decay, on the generalization performance of SVRG, and verify the generalization performance of Batch-SVRG (B-SVRG). In terms of the relationship between optimization and generalization, we believe that the average norm of gradients on each training sample as well as the norm of average gradient indicate how flat the landscape is and how well the model generalizes. Based on empirical observations of such metrics, we perform a sign switch on B-SVRG and derive a practical algorithm, BatchPlus-SVRG (BP-SVRG), which is numerically shown to enjoy better generalization performance than B-SVRG, even SGD in some scenarios of deep neural networks.

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Wide Residual Networks
• Training Very Deep Networks
• SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Deep Networks with Stochastic Depth
• Stochastic Variance Reduction for Nonconvex Optimization
• Non-convex Finite-Sum Optimization Via SCSG Methods
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• On the Ineffectiveness of Variance Reduced Optimization for Deep Learning
• Data-Dependent Path Normalization in Neural Networks