An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise

Abstract

The choice of batch-size in a stochastic optimization algorithm plays a substantial role for both optimization and generalization. Increasing the batch-size used typically improves optimization but degrades generalization. To address the problem of improving generalization while maintaining optimal convergence in large-batch training, we propose to add covariance noise to the gradients. We demonstrate that the learning performance of our method is more accurately captured by the structure of the covariance matrix of the noise rather than by the variance of gradients. Moreover, over the convex-quadratic, we prove in theory that it can be characterized by the Frobenius norm of the noise matrix. Our empirical studies with standard deep learning model-architectures and datasets shows that our method not only improves generalization performance in large-batch training, but furthermore, does so in a way where the optimization performance remains desirable and the training duration is not elongated.

References in corpus (33)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• The Loss Surfaces of Multilayer Networks
• Don't Decay the Learning Rate, Increase the Batch Size
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Train faster, generalize better: Stability of stochastic gradient descent
• New insights and perspectives on the natural gradient method
• No More Pesky Learning Rates
• Adding Gradient Noise Improves Learning for Very Deep Networks
• How Does Batch Normalization Help Optimization?
• Three Factors Influencing Minima in SGD
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• Characterizing Implicit Bias in Terms of Optimization Geometry
• Measuring the Effects of Data Parallelism on Neural Network Training
• Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Geometry of Optimization and Implicit Regularization in Deep Learning
• Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints
• A Walk with SGD
• Three Mechanisms of Weight Decay Regularization
• Understanding Short-Horizon Bias in Stochastic Meta-Optimization
• Stability and Convergence Trade-off of Iterative Optimization Algorithms
• Efficient Per-Example Gradient Computations
• On the diffusion approximation of nonconvex stochastic gradient descent
• A Surprising Linear Relationship Predicts Test Performance in Deep Networks
• A Coordinate-Free Construction of Scalable Natural Gradient

Cited by in corpus (7)

• Don't Use Large Mini-Batches, Use Local SGD
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• A Study of Gradient Variance in Deep Learning
• On Accelerating Distributed Convex Optimizations
• Contrastive Weight Regularization for Large Minibatch SGD
• Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD